Visualizing geographic-area change detected from high-resolution, remotely sensed imagery

ABSTRACT

A method, system, and medium are provided for presenting aspects of change associated with a geographic area that has been captured by high-resolution, remotely sensed imagery. One embodiment of the method includes receiving a query directed at the geographic area that includes one or more inputs, the query seeking an identification of regions associated with the geographic area that are characterized by aspects of change based on the one or more inputs; applying the query to a dataset of geospatial information that stores imagery associated with the geographic area, wherein the dataset includes information that is sufficient to identify the regions, and wherein the imagery is derived from the high-resolution, remotely sensed imagery, which is characterized by having a resolution of three meters or less per pixel; receiving a first results set that includes a first plurality of keys and corresponding change scores, wherein, (1) each key is useable to identify a certain region, and (2) each change score indicates an amount of change in the certain region from a first state to a second state; and presenting at least a portion of the results set in a viewing application.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/106,803, filed Oct. 20, 2008 and U.S. Provisional Application No.61/105,955, filed Oct. 16, 2008, which are both expressly incorporatedby reference herein in their entirety.

Also, much of this disclosure is shared with the following U.S. patentapplications, filed on even date herewith, one of which is thisapplication:

Title Attorney Dkt. Filing Date DETECTING GEOGRAPHIC-AREA CHANGE USINGUNOM.144498 Oct. 20, 2008 HIGH-RESOLUTION, REMOTELY SENSED IMAGERYIDENTIFYING GEOGRAPHIC AREAS BASED ON UNOM.144499 Oct. 20, 2008 CHANGEPATTERNS DETECTED FROM HIGH- RESOLUTION, REMOTELY SENSED IMAGERY STORINGCHANGE FEATURES DETECTED FROM UNOM.144500 Oct. 20, 2008 HIGH-RESOLUTION,REMOTELY SENSED IMAGERY VISUALIZING GEOGRAPHIC-AREA CHANGE UNOM.144501Oct. 20, 2008 DETECTED FROM HIGH-RESOLUTION, REMOTELY SENSED IMAGERYREVISING IMAGERY SEARCH RESULTS BASED UNOM144502 Oct. 20, 2008 ON USERFEEDBACK

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

This invention was made with U.S. Government support by way of GrantNumbers HM1582-04-1-2028 and HM1582-6-C-0018 awarded by the NationalGeospatial Intelligence Agency. The Government has certain rights in theinvention. See 35 U.S.C. §202(c)(6).

INTRODUCTION

Generally, this disclosure (patent/patent application) is directed tosystems and methods for processing geospatial information. Ourtechnology includes many facets that can be used in a variety ofsettings. In one illustrative setting, our technology helps locate areasof change in a given geographic area. Our technology facilitatesautomatic change detection (at the region or item level), which is ableto utilize high-resolution remotely sensed imagery to detect changesassociated with a geographic area. The remotely sensed imagery might beobtained by way of a satellite, an airplane, a UAV or other method.

An illustrative use of our technology, as will be expanded upon below,might be, for example, to detect the presence of a new road that hasrecently appeared over some period of time. Perhaps this road wasconstructed over the course of a year in a remote geographic area. If aperson were armed only with images that depicted the area at variousstates, the individual might not know where to even begin looking forthe road. That is, assume that a satellite photographed a given area ofland. This given area of land (not to exclude water) might be vast, suchthat an individual would not even know where to start looking.

Moreover, even if an individual did know where to look, the individualmight only see the constructed road, which might not be relevant becausethe road might have been present for any number of years (assuming thatthe individual is interested in finding areas that have changed overtime). Our technology enables an automatic detection of a feature suchas the recently constructed road. It enables an automatic analyzing andcomparing imagery associated with a given geographic area in multiplestates to identify aspects of change associated with that geographicarea. Still further, other constraints unrelated to the imagery can alsobe levied by way of a query so that even more specific items of interestcan be found. For example, one might wish to search for all rectangularobjects within a certain distance of a school within a given area.

Still further, our technology enables not only a searching for objects,as well as identifying change associated with objects, it also enablesan identification of geographic areas that exhibit changecharacteristics that are similar to a provided set of changecharacteristics. For example, consider a desert. Now assume that a roadhas been constructed in the desert. This geographic area has changed ina certain way over a certain amount of time. Now, assume that a userwishes to locate other geographic areas that have exhibited similarchange. For example, assume that a user wishes to locate other roadsthat have been constructed in a desert. This is not the same as merelysearching for roads in a desert, for that would only take into accountan “after” state; that is, that search would only find roads in deserts.But what our technology allows is a location of a geographic area thathas changed in a manner similar to an indicated type of change. Thus,for example, only roads in deserts that have recently been constructedcan be returned as a result incident to receiving a query in oneembodiment of our technology.

We are providing very high level examples to help introduce concepts ofour technology before we begin describing it in detail. Although wementioned roads by way of introduction, it will become clear that manytypes of features common, including change in general, can be identifiedby employing aspects of our technology. Generally, our technology isemployed in connection with high-resolution remotely sensed imagery. Oneform of this includes commercial satellite imagery (CSI). Generally,when we speak of “high-resolution” we are speaking of those resolutionsthat have a ground sample distance (GSD) of 3 meters or less; that is, 3meters or less per pixel. Processes described herein can utilize imageryon the order of one meter per pixel, 0.6 meters per pixel, or evenhigher resolutions.

Our technology includes storing change-detection information, andmoreover, storing it in such a manner and in such a data structure thatscaling is efficient. Our data is stored in such a way that it lendsitself to high scalability and efficient organization as well assearching. We will describe ways of fusing together results from severalindividual feature indexes to ultimately rank tiles that depictdifferent geographic areas. This automatic ranking enables a user torapidly identify areas of interest and depict them on a viewer, which,as we will describe in much greater detail below, enables the identifiedgeographic areas to be viewed in context, among their surroundings. Thisenables a user's eye to be quickly directed to relevant areas instead ofhaving to laboriously, and often fruitlessly, search a mass ofinformation. In many situations today, a great deal of remotely sensedimagery sits idle, unsearched, and unused because there is simply toomuch information to even start analyzing. But aspects of our technologyenable not only a beginning process to analyze such imagery, but also amethodical storing, analyzing, rapidly searching, and visual depictionof relevant data within the massive imagery.

SUMMARY

We describe various technological aspects associated with processinghigh-resolution satellite imagery.

The claims define embodiments of our invention, not this summary. Thissummary is provided to introduce a selection of concepts in a simplifiedform that are further described below in the Detailed Description. ThisSummary is not intended to identify key features or essential featuresof the claimed subject matter, nor is it intended to be used to limitthe scope of the claimed subject matter. In brief, we describeillustrative ways of visualizing aspects of change associated with ageographic area that has been captured by high-resolution remotelysensed imagery.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Illustrative embodiments of our technology are described in detail belowwith reference to the attached drawing figures, which depict certainembodiments, which figures are incorporated by reference herein, andwherein:

FIG. 1A depicts a first illustrative operating environment suitable forutilizing the described technology;

FIG. 1B depicts two illustrative geographic areas along withcorresponding imagery;

FIG. 2 depicts an overall process flow that depicts various aspects ofour technology;

FIG. 3A depicts an illustrative image-ingestion process;

FIG. 3B depicts an illustrative data structure for cataloging imagery;

FIG. 4A depicts illustrative co-registration processes;

FIG. 4B depicts illustrative regions and corresponding imagery that areused to help explain an illustrative co-registration process;

FIG. 4C depicts aspects of a co-registration process that co-registersvarious scenes with each other;

FIG. 4D depicts a co-registration process where images are co-registeredto a common base scene;

FIGS. 5A-5D depict illustrative processes of extracting features fromimagery;

FIGS. 5E-5G depict illustrative aspects associated with calculating achange-quality measurement;

FIG. 6A is a block diagram illustrating aspects of an exemplarychange-detection process;

FIG. 6B depicts an illustrative change-detection process;

FIG. 6C depicts an illustrative set of feature bands and various valuescorresponding to various pixels;

FIG. 6D depicts an illustrative method for quantifying change associatedwith regions;

FIG. 6E depicts an illustrative feature vector;

FIG. 6F depicts a high-level overview of calculating a difference vectorby differencing a source feature vector and a target feature vector;

FIGS. 6G-6I depict embodiments for determining values of a targetpixel-level feature vector;

FIG. 6J depicts an illustrative process for refining a change-confidenceimage;

FIG. 6K depicts an illustrative histogram associated with anillustrative feature vector;

FIG. 7A depicts an illustrative data model for storing imagery featuredata;

FIG. 7B depicts a high-level overview of an illustrative process forstoring imagery feature data;

FIG. 7C depicts how a key might be associated with various informationalattributes of a given geographic area and how information can beseparately indexed based on separate features;

FIG. 7D depicts an illustrative multidimensional feature vectors;

FIG. 7E depicts illustrative feature bands and corresponding dimensionsassociated with each;

FIG. 7F depicts an embodiment for fusing together results associatedwith searching separate feature indexes to ultimately arrive at anoverall result;

FIG. 8A depicts an illustrative method for retrieving results based on areceived query;

FIG. 8B is a block diagram that depicts various types of query inputsthat might be associated with a given query;

FIGS. 8C-8E depict more illustrative flows associated with searching forrelevant results;

FIG. 8F depicts illustrative relationships between keys and rankingscores;

FIGS. 8G-8I depict illustrative processes for applying feedbackindications to a set of results;

FIGS. 9A-9C depict illustrative methods for automatically detectingchange associated with a geographic area;

FIGS. 10A-10C depict illustrative methods for identifying terrestrialobjects that have changed in a certain manner;

FIG. 11A depicts a set of illustrative user-input options that can beused to form a query;

FIGS. 11B-11D depict aspects of returning results associated with aregion-based query;

FIG. 12A depicts a state diagram that illustrates how multiple viewerscan be updated based on input from other viewers;

FIG. 13 depicts an illustrative user interface for receiving query-inputoptions associated with seeking change information associated with oneor more geographic areas;

FIGS. 14A-14B depict an illustrative user interface for presentingresults associated with a change search query;

FIG. 15 depicts additional results of the set depicted in FIG. 14A;

FIG. 16 depicts another way of presenting returned image tiles;

FIGS. 17A-17B depict aspects associated with change information forvarious geographic areas;

FIG. 18 depicts geographic regions in a grid-like pattern as well ascorresponding user-feedback options;

FIG. 19 depicts geographic areas as well as corresponding user-feedbackoptions;

FIG. 20 depicts a set of revised results that were revised based on thevarious user-feedback options of FIG. 19;

FIG. 21 depicts an illustrative imagery viewer;

FIG. 22 depicts a set of tiles in an imagery viewer in one embodiment;

FIG. 23 depicts a subset of some of the tiles (2222) from FIG. 22 ingreater detail;

FIG. 24 depicts an illustrative control option that presents exemplaryhyperlinks usable to interact with the various tiles presented;

FIG. 25 depicts the results of utilizing an embodiment of the “smartsurround” feature of FIG. 24;

FIG. 26 depicts tiles similar to those of 25, but perhaps in a differentstate;

FIG. 27 depicts additional tiles incident to acting on the “smartsurround” option of FIG. 26;

FIG. 28 depicts illustrative controls that are usable to interact withour technology to facilitate a tile-animation sequence;

FIGS. 29A-29B depict illustrative processes for visualizing searchresults; and

FIGS. 30A-30C depict illustrative methods for revising a set of searchresults based on feedback provided.

DETAILED DESCRIPTION

Throughout the description of our technology, several acronyms,shorthand notations, and terms are used to aid the understanding ofcertain concepts pertaining to the associated system and services. Theseacronyms and shorthand notations are intended to help provide an easymethodology of communicating the ideas expressed herein and are notmeant to limit the scope of our technology unless we clearly say so. Thetable below does not include a full definition. A full definition of anyterm can only be gleaned by giving consideration to the full breadth ofthis patent.

Sometimes, we use different terms to refer to similar or the samethings. We do not necessarily mean to implicate a difference per se; butare constrained by certain statutory requirements that apply to patentsand patent applications. For example, claims must use proper antecedentbasis. Sometimes, satisfying this rule can lead to wordiness, whereasusing a different word helps make referring to prior terms easier. Thus,we might use different words in that regard. The following is a list ofthese terms:

Term Short Reference chip image portion reflected in a viewerapplication destination/target/ used interchangeably and generally,refers after to a subsequent thing; e.g., “an after image” DMPDifferential Morphological Profile EBB Entropy Balanced Bitmap EBSEntropy Balanced Statistical geographic used interchangeably with“terrestrial” to refer to the surface of a body, such as a planet, andis not intended to be limited to a land context (to the exclusion ofwater) GSD Ground Sample Distance high resolution Generally, resolutionsat a GSD higher than about 3 m; that is resolutions of 3 meters or lessper pixel (e.g., 1 m/pixel and fractions of meters per pixel) imagegenerally, a picture as well as metadata associated with the picture,though distinguishing from “picture” is not intended imagery acollection of images or image data KML Keyhole Markup Language MSMultispectral NDVI Normalized Difference Vegetation Index NIRNear-Infrared Pan Panchromatic region generally, used interchangeablywith “scene” and refers to all or a portion of an image; sometimes ageographic area RSI Remotely Sensed Imagery (imagery received by way ofsatellites or aircraft for example) scene generally, usedinterchangeably with “region” and refers to all or a portion of an imagesource/reference/ used interchangeably and generally, refers before toan initial or base thing, e.g., “a before image” subscene generally, aportion of an image, used interchangeably with “tile” or “image portion”terrestrial used interchangeably with “geographic” to refer to thesurface of a body, such as a planet, and is not intended to be limitedto a land context (to the exclusion of water) tile generally, a portionof an image, used interchangeably with “subscene” or “image portion”;sometimes used synonymously with “chip” when referring to an image in aviewer; (can be logical, not necessarily individually stored) UAVUnmanned Aerial Vehicle UTM Universal Transverse Mercator

Regarding the use of singular and plural, we do not mean to intend anysort of strict numerical implication by using the singular or plural ofa term; similar to our lack of intent to imply the singular by using “a”or “the.” Trying to capture the plural in words or in the FIGs. wouldoften lead to wordiness. For example, though we might refer to “adatabase,” clearly we fully anticipate that such reference would beequally applicable to multiple databases. By way of another example “amemory” does not imply one single memory device.

We do not intend for inferences to be made by way of negativeimplication. That is, just because we say that some certain thing mightbe optional, for example, we don't mean to imply that anything notmentioned as optional is required. Also, we do not mean to imply anactual first in a sequence when we use the term “first.” A first doesnot mean the first. We are simply referring to a particular thing, anduse the terms “first,” “second,” and the like to keep things separateand make referencing easier.

Embodiments of our technology may be embodied as, among other things: amethod, system, or set of computer-useable instructions embodied on oneor more computer-readable media. Computer-readable media include bothvolatile and nonvolatile media, removable and non-removable media. Byway of example, and not limitation, computer-readable media comprisemedia implemented in any method or technology for storing information.Examples of stored information include computer-useable instructions,data structures, program modules, and other data representations. Mediaexamples include, but are not limited to information-delivery media,RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM,digital versatile discs (DVD), holographic media or other optical discstorage, magnetic cassettes, magnetic tape, magnetic disk storage, andother magnetic storage devices. These technologies can store datamomentarily, temporarily, or permanently.

Turning now to FIG. 1A, an illustrative operating environment suitablefor practicing an embodiment of our technology is provided andreferenced generally by the numeral 100A. A satellite 110A is showntaking an image 112A of a first geographic area 114A at a first time116A. The first time 116A is indicated as “T1.” The imagery 112A isdoubly represented by the remotely sensed imagery item having referencenumber 118A. This is an example that might occur more than once in thisdisclosure where different numerals do not necessarily implicatedifferent items. As previously mentioned, we are somewhat constrained byvarious technical requirements of a patent. Therefore, we will usemultiple numerals to refer to a potentially common physical item to helpthe reader follow along when referencing the drawings and reading thisspecification. Remotely sensed imagery (“RSI”) includes pictures 120A aswell as metadata 122A. Even though we use the term “pictures” separatefrom “metadata,” we do not mean to imply that other references topictures in this disclosure do not include metadata. Again, to helpexplain aspects of the invention, we are conveying that metadata 122A isincluded as part of remote imagery 118A. We will refer to RSI variouslyas “imagery,” “remote imagery,” or variations thereof.

FIG. 1A also depicts an airplane 124A as an illustrative vehicle forcapturing aerial photography. At a second time, 126A, a secondgeographic area 128A is captured. Note that an overlapping region 130Aintersects areas 128A and 114A. The imagery associated with geographicarea 128 becomes part of what is schematically shown as remotely sensedimagery 118A. RSI 118A could include a multitude of imagery received bythe same or other source and is generally shown here to include a meretwo illustrative examples. In fact, RSI 118A might take the form ofseveral gigabytes, terabytes, or even petabytes of imagery information.

As will be explained, a processing step 132A processes the remotelysensed imagery 118A resulting in modified imagery being stored as partof a dataset 134A. Of course dataset 134A might actually be severaldatasets across several types of computer-readable media, but it'sdepicted as shown so as to not obscure other aspects of the technologywe are describing. Dataset 134A provides information upon which tosearch incident to receiving a query 136A. In some embodiments, query136A is received by way of user interface 138A, which interacts withdataset 134A to provide a set of results 140A. Generally, results 140Ainclude a list of geographic areas that are relevant based on query136A. These results can be refined by way of user feedback or otherinputs, as will be explained below.

Turning now to FIG. 2, a somewhat more detailed flow chart is providedthat depicts an illustrative method for practicing various aspects ofour technology. We make reference to other figures in FIG. 2, and do soto help the reader, but do not intend such other references to mean thatthese are the only places in this disclosure that a certain aspect isdiscussed. Moreover, although it should go without saying, we do notmean to convey that certain processes have discrete starting andstopping points. That is, we do not mean to imply that animagery-ingestion process starts or stops precisely where shown, nor aco-registration process, nor any process shown. In reality, the processboundaries, to the extent they exist at all, are fuzzy, gray areasrather than defined by crisp black and white lines. Again, we are merelytrying to provide the reader with an illustration that helps conveyvarious aspects of our technology. We do not mean to limit ourtechnology beyond what we say. Moreover, those of ordinary skill in theart might refer to processes that we refer to herein by different names.What one calls an “ingestion process” another might refer to as animage-acquiring process. Further, what one calls an image“co-registration process” another might refer to as a scene-matchingprocess. We do not mean to convey by our terms specific processes butneed to use some terms in order to even discuss the technology.

A set of raw imagery 210 serves as input to an imagery-ingestion process212. Raw imagery 210 might be remotely sensed imagery 118A of FIG. 1A inone embodiment. In other embodiments, raw imagery might either not bereceived or start off as what we refer to later as modified imagery.That is, in some embodiments, imagery 210 is truly at a raw state,meaning untouched or uncorrected; but in other embodiments, it takes theform of initial imagery that might have already been subjected to one ormore image-correction processes.

After imagery 210 is ingested at a step 212, features associated withthe ingested imagery are extracted at 214. The types of features thatcan be extracted at step 214 can either be region based (that is, areabased), or pixel based. In one embodiment, pixel-level features areextracted at this point for the whole area that imagery 210 relates to.In other embodiments, features at the region level only are extracted.We will describe later that generally, region level features representaggregations of pixel-level features. In one embodiment of ourtechnology, pixel-level features are utilized to facilitate changedetection, which is only implicated when overlapping imagery is present;that is, multiple sets of imagery related to the same geographic areabut at different times. Thus, in one embodiment, pixel-level featuresare stored after a determination is made that overlapping imageryexists; but, as mentioned, in other embodiments, pixel-level featurescan be stored initially.

Dashed line 216 indicates that output from the image-ingestion process212 can take the form of a reference scene 218 as well as a target scene220. Reference scene 218 is a depiction of a geographic area at a first,or “reference,” state. Reference scene 218 includes correspondingmetadata 222. Target scene 220 also includes metadata, 224, and depictsthe same geographic area but in a second state. In some embodiments,reference scene 218 was previously stored, and target scene 220, wasrecently stored. In other embodiments, both reference scene 218 andtarget scene 220 were processed simultaneously and recently stored. Uponstoring the imagery associated with target scene 220, reference scene218 is identified as imagery of the same geographic area as that oftarget scene 220; hence overlapping imagery.

Reference scene 218 and target scene 220 can provide inputs to aco-registration process 226. We will describe co-registering the scenesin greater detail below. A first output of co-registration process 226is received by a feature-extraction process 228. If, certain featureswere not extracted at step 214, they will be extracted here. On-pagereference A, referred to by numeral 230, indicates that the featuresextracted from process 214 and/or 228 are stored as part of one ormultiple storing processes, which are generally referenced by numeral232.

Another outflow from co-registration process 226 includes visualizationimagery 234. Visualization imagery 234 refers to imagery that has beenoptimized for presentation in a viewer application. In some embodiments,imagery might be 11 bits for example, whereas 8-bit imagery support ismore pervasive in imagery viewing applications, for example, webbrowsers. Accordingly, in one embodiment, 8-bit visualization imagery iscreated so that it can easily be recalled and sent to a viewerapplication on demand. In other embodiments, visualization imagery 234can be generated on the fly and presented in a viewer application. Asshown, visualization imagery 234 includes a reference visualizationscene 236 as well as a target visualization scene 238. In thisdisclosure, we will generally use the term “reference” to refer to afirst or initial state. We will generally use the term “target” to referto either a subsequent state of an object or area or to refer to a setof objects that are sought by way of a query. For example, if the useris seeking a specific object, we might refer to that object as a targetobject.

Also shown in FIG. 2 is a small item having the initials “CQM” that isreferred to by reference numeral 240. Reference numeral 240 refers to achange-quality measure. CQM 240 does not necessarily have a strongrelationship to visualization imagery 234. Rather, it is illustrativelyshown in this context to convey that it is associated with two or morescenes that correspond to a given geographic area. CQM 240 is achange-quality measure. That is, it measures a suitability of comparisonof a reference scene and a target scene. We will explain this in greaterdetail below, but in brief, one aspect of our technology seeks tominimize detecting change that is not of the sort that indicates whatmight be referred to as true change or real change. For example, asnow-covered ground doesn't necessarily reflect an actual change in theground per se. This is what might be referred to as seasonaldifferences. Other issues with analyzing imagery includes accounting forshadows, sensor geometry, sun position, and the like. CQM 240 accountsfor these various effects and quantifies a suitability that some targetscene could be compared against some reference scene. This data can bestored by way of process 232.

Returning to feature-extraction process 228, processing advances to achange-detection process 244. The change-detection process utilizes,among other things, the features previously extracted from the imageryto detect change between two or more scenes, resulting in changeinformation 246 (variously referred to herein as “change-detectioninformation”). Another outflow from the change-detection process 244includes what we refer to as a change map 248. A change map, as willalso be explained, provides a visual indication of both an intensity andareas of change within a region. This is different than merelyindicating whether a given pixel has changed or not, a binary-typesolution. To the contrary, change map 248 is somewhat akin to a weathermap or a radar map. A storm depicted on a radar map has certain degreesof intensity that are reflected by color such that areas of greaterdisturbance are color-coded and easily visually perceptible.

Our technology is, of course, very different. But in one embodiment, thechange map that a user sees looks somewhat akin to such a radar map inthat areas of change are color-coded (or grayscale coded) and shadedsuch that areas of intense change are contrasted against areas ofrelatively small change. In one embodiment, change map 248 can besubdivided into a set of subscenes 250. Dividing an area into subscenescan also be done in connection with the visualization imagery. What wemean here is that a scene might be of a large geographic area, whichmight be difficult to present on a viewer. Accordingly, a scene might besubdivided into smaller regions, or what we might refer to herein astiles. By way of illustration, a tile might correspond to a geographicarea on the order of 256 square meters. Of course, this unit of area isnot meant to be limiting, but is provided by way of illustration. Thisinformation is stored by way of data-storage process 232.

An outflow from data-storage process 232 includes a dataset 252. Dataset252 might actually be what some refer to as multiple sets of data. Byway of illustration, we indicate that dataset 252 might include imagerydata 254 as well as other geospatial data 256. Imagery data 254 mighttake the form of the features and change information that werepreviously extracted in processes 214, 228, 244, and 246 as well asinformation derived from interacting with the imagery 210.

Other geospatial data 256 includes data from heterogeneous databases;that is, databases that stored disparate types of information. Forexample, a first type of information could include areas of interest,such as fun parks or historical sights. Other types of information mightinclude information related to military institutions, or other buildingssuch as schools and churches. We will mention more examples later, theyare almost without limit, and these provide a way to include additionalconstraints of a query 258 that seeks certain types of information.

Query 258, as with almost all of these components, will also bedescribed in greater detail below. Briefly, it can include multipleinputs seeking geographic areas that exhibit certain traits; forexample, as having certain features, or having exhibited a certainchange signature, and more.

Illustrative viewers include an image viewer, a mapping component, a webpage, and more, as will be discussed in greater detail below. Imageviewers can include virtual-globe viewers like Google Earth, Microsoft'sVirtual Earth, ESRI's ArcGlobe and others.

Turning now to FIG. 3A, an illustrative method for ingesting remotelysensed imagery is provided in reference generally by the numeral 300. Ata step 310, the remotely sensed imagery is cataloged. In one embodiment,the remotely sensed imagery of step 310 includes remotely sensed imagery118A of FIG. 1A. Imagery 118A may be obtained in a variety of ways. Inone embodiment, the imagery is communicated from satellite 110 or airvehicle 124 directly. In other embodiments, imagery 118A arrives by wayof a data store on one or more computer-readable media. Cataloging theimagery includes storing the imagery in a database so that it can beaccessed. The metadata is also stored in the database such that theimagery can be referenced and operated on. Illustrative metadataincludes an image date, source information, calibration data,orientation data describing the geometry at which a picture is taken, alatitude and/or longitude indication, a UTM indication, or othergeo-reference location information, and the like. Imagery can bereceived in multiple resolutions. For example, a first set of imagerymay be received at a resolution of 0.6 meters while other imagery isreceived at a resolution of one meter. By way of illustration, anillustrative scene might include an area on the order of tens orhundreds of square kilometers. Illustrative formats of imagery 118Ainclude GeoTIFF and NITF. Sometimes multiple images are associated withwhat we refer to as a scene. The multiple images might be imagesassociated with different wavelengths. For example, a first image mightbe a certain spectral band while another image might be of the samegeographic area but in the near-infrared (NIR) band.

Turning briefly to FIG. 3B, an illustrative data structure associatedwith cataloging remote imagery 118A is provided and referenced by thenumeral 314. Those skilled in the art will appreciate other ways ofcataloging imagery 118A as well as alternative data structures. Datastructure 314 is provided by way of illustration.

Returning to FIG. 3A, reference numeral 312 refers to the start of anillustrative ingestion process. Process 312 could operate on multiplesets of imagery, depicting the same or different regions, at the same ordifferent times. The illustrative flow 312 will describe an example ofingesting a set of new imagery.

At a step 316, raw imagery 118A is modified to derive modified imageryif applicable. As previously mentioned, in some cases it might not benecessary to modify the imagery received at step 312. In otherembodiments, raw imagery 118A might benefit from being modifiedaccording to an illustrative process as we will describe. We refer tothe modified imagery as ingested imagery. An illustrative modificationprocess 316 is shown by way of reference numerals 318-326. At a step318, imagery 118A is resampled if applicable. This is one example of away to deal with imagery at multiple resolutions. In one embodiment,imagery is resampled to a nominal resolution. As is almost always thecase with the various processes described herein, these steps do notneed to be performed in the order shown. Generally, some order must beshown in order to be able to discuss the various steps, but one shouldnot draw from the order in which the different steps of the variousprocesses of all the different flow charts appear that the steps need toappear in the order shown.

At a step 320, geometric distortions in the imagery are corrected. Thisgeometric-correction process may be useful when the 3D topography of ageographic area causes distortions of the 2D pixel locations within theimagery. Thus, one way of facilitating the geometric-correction processof step 320 is to reference a set of information that describeselevation data, such as a Digital Elevation Model (DEM), that isassociated with the geographic area. This data can be used along withother metadata, such as sensor-pointing geometry, to correct forgeometric displacements in the pixels associated with the geographicarea's topography. For example, a given hill might appear in a differentimage location in a first scene than in a second scene because the firstscene was derived from a camera that took a picture of a hill at a firstangle while the same hill that appears in the second scene was shot at adifferent angle. Because the hill has height, taking pictures of thehill at different angles might make the hill appear in differentlocations in the two scenes even though it has not changed at all. Ifthe elevation aspects were not accounted for, then during the changedetection process, false positives could be presented because ouralgorithms would detect change in imagery that does not reflectreal-world change. It could show change when there actually was none.Thus, in one embodiment, pixel locations in the scene are relocated tocorrect for the 3D topography.

At a step 322 the imagery is re-projected. Re-projection includesre-projecting to a common coordinate system such as UTM, state plane,geographic (latitude/longitude), military-grid-reference systems, or anyother coordinate system. The general idea is to re-project onto acoordinate system that is the same as the coordinate system that priorimagery is mapped to so that a good comparison can be made.Re-projection can also help mitigate the effects of a planet'scurvature.

At a step 324, reflectance values are derived from the sensor's digitalnumbers (DNs) that represent the intensity of the pixels that make upthe imagery. For many analysis techniques that utilize spectral data inthe imagery, it is beneficial to convert the sensor DNs to reflectancebefore further processing. This is useful to correct for various sensorcharacteristics that cause fluctuations in the DNs that are not due toreal-world changes in geographic area. For example, exposure settingscan be part of the metadata associated with the imagery and the exposurelevel may vary between a first scene and a second scene. That is,exposure settings across imagery taken at different times might not bethe same. In order to accommodate for this potential discrepancy, anillustrative solution is to convert pixel values to reflectance usingsensor calibration information and other information from the imagemetadata. Then, the reflectance values may be used when comparing pixelsin subsequent change detection procedures.

A result of completing process 316 includes producing ingested imagery328. This ingested imagery, which we also refer to as modified imageryif it was modified, can serve as inputs to co-registration process 330as well as feature-extraction process 332. As previously mentioned, wemight use different reference numerals to refer to the same process.Thus co-registration process 330 might be the same as co-registrationprocess 226 of FIG. 2. But by using unique reference numerals, it iseasier to refer to the various components of the drawings.

Ingested imagery 328 can serve as input to co-registration process 330if any or all portions of ingested imagery 328 overlaps with previouslystored imagery. Or, in the case of an on-demand embodiment, two scenescan be compared in real time and, to the extent they are modified, canprovide input to co-registration process 330.

Ingested imagery 328 might serve as input to feature-extraction process332 if features are to be extracted at this time. As previouslymentioned, region and/or pixel features can be extracted at this time.In one embodiment, region type features are extracted. That is, featuresassociated at a region or tile level. This does not need to be done tothe exclusion of extracting pixel-level features, which can also be donein one embodiment. That is, at this stage pixel-level features can beextracted from all imagery ingested irrespective of whether that imageryoverlaps with previously stored imagery. Extracting pixel-level featuresat this stage would avoid having to extract the pixel-level features ata later stage. But if one desires to conserve storage space, thenextracting pixel-level features can wait until a determination thatoverlapping imagery exists.

Turning now to FIG. 4A, an illustrative co-registration process isprovided in reference generally by the numeral 400. At a step 410,imagery is queried to identify overlapping scenes. In one embodiment,metadata is used to roughly determine whether overlapping imageryexists. The imagery queried can be that stored in catalog 314 in oneembodiment. Two illustrative processes for performing co-registrationare provided in reference by numerals 412 and 414. As briefly mentioned,co-registration involves lining up images of a geographic region. Toproperly detect change, two images of the same area should be properlylined up so that corresponding pixels can be compared.

Consider a first image that depicts a given geographic area in a firststate. If a second image depicts the same geographic area in a secondstate, then co-registration will result in the second image being linedup with the first image. We use the term “line up” loosely because whatactually happens is that each pixel in a target image is run through atransformation algorithm so that it is mapped to a location thatcorresponds to a sister pixel in the source image. Thus one can think ofco-registration as co-registering a target scene to a source scene. Buta question arises as to what should serve as the base scene. In oneembodiment, the base scene is simply the first image. This process isdescribed by numeral 412. In a second embodiment, the base scene is somecommon scene that includes both scenes. This latter embodiment isillustrated by reference numeral 414. It might be the case that a commonscene could be as large as the entire earth. That is, to the extent itis available, imagery that includes both scenes, or an entire set ofscenes, can be used as a common reference image upon which to base aco-registration process for two or an entire set of scenes.

Turning to the process 412 for performing co-registration using apermutation of scenes, potential tie points are extracted for eachtemporal pair of overlapping scenes at a step 416. A temporal pair ofoverlapping scenes is made up of two scenes of the same geographic areataken at different times.

Tie points can be extracted using various methods. A first method thatmight be employed is a scale-space method that selects points with localminimum or maximum intensities that persist across different imagescales. Other illustrative methods that might be employed include imagecross-correlation, corner feature detection, and mutual information. Tiepoints are points selected in the images that will be used as referencepixels that, when lined up, will result in the entire images being moreclosely lined up. More technically, all image pixels in a target sceneare subjected to a transform that alters their locations, therebyresulting in a target scene with pixels that properly, or at leastapproximately, correspond to matching pixels in the source scene.

At a step 418 we determine good tie points from the initial set ofpotential tie points. Not all tie points extracted in step 416 are ashigh a quality as others. The better the set of tie points, the moreaccurate the lining up of two images will be. We use several methods todetermine the good set of tie points. In one illustrative method, wemeasure a pattern corresponding to the intensity variations of pixelssurrounding each tie point. These patterns are compared between sourceand target tie points to find points whose patterns are a good match.Another illustrative way of determining good tie points is to calculatemultiple instances of the same image transform, like global affine inone embodiment, from many individual subsets of the potential tiepoints. Parameters like x-translation and y-translation from themultiple global affine transforms can be analyzed to determine how wellthey agree. Good sets of tie points often produce similar transformparameters, while bad sets of tie points do not. Thus, in one embodimentbad tie points can be identified in this way and removed.

At a step 420, multiple transform algorithms are applied to determinewhich algorithm of the multiple algorithms should be used to transformthe entire target image. The one with the least co-registration errorwill be used. Three illustrative transforms that can be employed includeglobal affine, Delaunay triangulation, and thin-plate spline. Each ofthese algorithms can be applied using an initial subset of good tiepoints from step 418. The remaining good tie points, that is those notused in the calculating the transforms, are used to independentlymeasure the resulting co-registration error. This is repeated many timesusing different sets of initial (used in the transform) and remainingtie points (not used in the transform) to statistically verify theco-registration error. The algorithm that generates the smalleststatistically verified co-registration error is selected. This algorithmis then applied to the entire target scene at step 422 to derivetransformed imagery.

We will return to FIG. 4A but now turn to FIG. 4B to describe theco-registration process in greater detail and give some examples. FIG.4B depicts three illustrative scenes, each referenced by the separateletters A, B, and C. Three overlapping regions exist: AB, AC, and BC. Ofthose regions, one region is common to all three scenes: ABC. In thisexample, without loss of generality, assume that pictures of region Awere taken earliest in time, then B, then C.

Imagery related to region AB is called out and referenced by numeral426. The two constituent images that serve as a basis for comparingimages of region AB include image A 428 and image B′ 430. Image B′ 430is indicated as having been modified from its original state because itwas subjected to a transformation algorithm so that it co-registeredwith image A 428. As previously mentioned, imagery such as 428 and 430might have information in it that prevents it from being ideal todisplay in its native format. In embodiments where the imagery isdesired to be presented in a way that humans can view it in a morevisually appearing manner, what we are referring to as “visualizationimagery” is created. Thus, visualization image 432 is a pretty pictureof image 428, and visualization image 434 is a pretty picture of image430. Although we have not yet described our change-detection processes,after those change-detection processes have been applied tocorresponding scene pairs, change-visualization imagery can also becreated. Change-visualization imagery is imagery that depicts changeacross two scene pairs. We describe that here as ΔAB visual 436.

We show another example in connection with region ABC 438. In thisexample, we have an image of the ABC region taken at a first time, whichis referenced by numeral 440, an image 442 of the same region taken at asecond time, as well as a third image 444 of the same region taken atstill a different time. If it is desired to be able to present changeinformation associated with all of the different variations of thegeographic area at all of the various times for which we have seeninformation, then co-registration will occur for each combination. Butas mentioned, the question arises as to what should serve as the baseimage for performing co-registration. The example that we are in themiddle of describing contemplates the method of 412, which isillustrated here. Three reference images are used and three targetimages are used. The reference images are indicated by numeral 446 andthe target images are referenced by numeral 448. Note that in thisexample, the reference images do not have primes, meaning they have notbeen subject to a transformation. But the target images 448 do haveprimes, indicating that they have been subject to a transformation byway of being co-registered with each respective reference image. Forexample, image B′ 450 has been subjected to a co-registration process inwhich its original pixels were transformed based on reference image A452. But image B′ 450 is not the same as image B 454 because, asmentioned, image B′ 450 has been transformed, whereas image B 454 hasnot.

By way of another example, consider image C′ 456. It is labeled ashaving one prime because it has been transformed to be co-registeredwith reference image A 458. But image C′ 456 is different than image C″460 because image 460 was not transformed by the same reference image.That is, C″ was derived from a transformation applied to it based onimage B 454, as opposed to image A 458. In this example, image A 452 andimage A 458 are the same image since it serves as the reference imagefor both B′ 450 and C′ 456.

Pressing this example further, we turn to FIG. 4C, which depicts anexample of ten scenes that all overlap a common region, which isreferred to as the D-M region 462. Without going through everypermutation, they illustratively show that image E needs to beco-registered to image D, resulting in E′ 464; image F needs to beco-registered with image D, resulting in F′ 466; and then every otherimage greater than the prior will need to be co-registered as well. Forexample, scene F will need to be co-registered with region E, which willresult in image 468 F″, which is different than F′ 466. Herein lies anissue. As indicated, a maximum of forty-five co-registrations will needto be made if it is desired to be able to depict change across allpermutations in this embodiment. The forty-five registrations are shownby brace 470. Moreover, arguably extra visualization imagery will needto be created and stored. For example, visualization imagery thatcorresponds to F″ 468 will need to be created, and this is depicted bynumeral 472, which represents a visualization image corresponding to F″468. The maximum data to store 474 would be all of that data that makesup the visualization imagery across all permutations of combinations ofscenes of a given geographic area taken at various times.

Now returning back to FIG. 4A, and with reference to numeral 414, analternative process is described in which co-registration is performedusing a common base scene. At a step 476, reference is made to a basescene, which can be pre-identified. This is a single scene. We willdescribe this example in connection with FIG. 4D. An illustrative basescene is referred to by numeral 478. Again, this base scene encompassesbut is not limited to all the regions D-M, which were introduced in FIG.4C. By way of illustration, a base scene might be a scene that depictsthe United States, wherein regions D through M represent ten imagestaken of the State of Missouri at ten different times. Returning to FIG.4A, a determination of good tie points is made at a step 480. These tiepoints form tie-point pairs between the base scene and a given targetscene to be transformed, or co-registered with. These tie points areused at a step 482 to line up the target scene to the base scene. Ifapplicable, only a portion is utilized. As was the case with step 422,multiple algorithms are applied to the sets of tie points at a step 484to determine the best algorithm that should be applied to the entiretarget scene, which occurs at a step 486. Incident to step 486, thattarget scene is subject to the transformation algorithm derived fromstep 484, resulting in a transformed target scene. Processing will thenadvance to feature-extraction process 488.

Returning to FIG. 4D, using base scene 478, each of D-M images isco-registered to the base image at a step 490. This results in a set oftransformed images 492. Note that each image includes only a singleprime. That is, image D′ 494 is only a single prime, indicating that ithas been transformed from its original image, which, is indicated inFIG. 4C by referencing image 496. An advantage here is that the maximumnumber of co-registrations necessary will correspond to the maximumnumber of corresponding temporal scenes. Here, there were ten scenes:D-M. Thus, in this embodiment, there is a maximum of only tenco-registrations that are needed as opposed to the forty-five (N(N−1)/2where N=10) co-registrations that were necessary in FIG. 4C, which wasthe method of process 412 depicted in FIG. 4A. The process depicted bynumeral 414 in FIG. 4A, which is briefly depicted in FIG. 4D, conveysthat much less visualization imagery will need to be stored becausefewer co-registrations are performed, resulting in fewer “after”visualization images being needed.

Some embodiments employ method 412, while others employ method 414. If abase image such as base image 478 is not available, then method 412 cancertainly be used. If an image such as base image 478 is available, thena method such as process 414 can be employed.

As mentioned, processing advances to a feature-extraction process in oneembodiment. Turning to FIG. 5A, we introduce an illustrativefeature-extraction process. To discuss this illustrative process withsome context, we will make reference to a source scene 510 as well as atarget scene 512. We will discuss extracting features from one or bothof these scenes. As illustratively shown, ingested imagery 514 can serveas an input to one or more feature-extraction processes 516. As part ofthis process or as a separate process, a change quality measurement(“CQM”) is determined at a step 518. Processing then advances to achange detection process 520.

Returning to feature-extraction process 516, we indicate that such aprocess might be made up of one or more processes including, forexample, performing pixel-based processing at a step 522, performingobject-based feature extraction at a step 524, and performingregion-based feature extraction at a step 526. Again, these processesare not defined by bright lines, and might be known as other names, andare not constrained to the examples that we show. We will turn ourattention to an illustrative pixel-based processing algorithm withreference to FIG. 5B. What we refer to as pixel-based processing iscircumscribed by dashed line 528. Process 528 optionally includescalculating an NDVI value in 530, which relates to a vegetation indexvalue. This example does not have to be done (not that others have to),but doing so produces a mask to determine what pixels are associatedwith a type of feature, such as vegetation. In one embodiment, the NDVIis derived by taking a ratio of the red and near-infrared bands.

Returning to features-extraction process 516, we indicate that such aprocess might be made up of one or more processes including, forexample, performing pixel-based processing out of step 522, performingobject-based feature extraction out of step 524, and performingregion-based feature extraction out of step 526. Again, these processesare not defined by bright lines, and might be known by other names, andare not constrained to those that we show. We will turn our attention toan illustrative pixel-based processing algorithm with reference to FIG.5B. What we refer to as pixel-based processing is circumscribed bydashed line 528. Process 528 optionally includes calculating an NDVIvalue, which relates to a vegetation index value. This does not have tobe done (not that others have to), but by doing so we are able tothreshold these NDVI values to produce a mask to determine what pixelsare likely associated with vegetation and which are not. Thisinformation can be used, in one embodiment, to eliminate or reduce thecontribution of these vegetation pixels to any change detection process.This may be useful to minimize the influence of agriculture or otherseasonal vegetation changes when trying to detect other types of changelike new construction. In one embodiment, the NDVI is defined by afunction that uses the red and near-infrared band data.

Thus, the NDVI is calculated at a step 530 and a blurring option can beapplied at a step 532 if desired. Performing what we refer to as a blurhelps eliminate noise, things like cars on a road or patches on a road.These are illustrative micro discontinuities that might cause analgorithm to prematurely stop because the algorithm might recognize amicro discontinuity (such as a car on a road), as an indication of theend of a feature such as a linear feature, rather than a blip on alinear feature. Thus, as we will explain later, some of our algorithmsemployed are used to detect linear features. One aspect of calculating alinear feature is to determine where the linear feature starts andstops. Applying this blur technique prevents an algorithm employed fromthinking that the road stops where the car starts. Applying this blurcomponent helps the system disregard the car as a discontinuity in theroad.

At a step 534 spectral features are extracted, resulting in a set ofspectral features 536 being derived. When we make reference to“spectral” features we are generally referring to aspects of color orother aspects of spectrum. Accordingly, at a pixel level, in oneembodiment, we analyze multi-band images and derive features associatedwith red, green, blue, near-infrared, black and white (panchromatic)values. In another embodiment, alternate color spaces could be used suchas HSV, HSL, or CMYK. In some embodiments, the NDVI value is alsoderived or modified in connection with performing spectral analysis. Aswill be explained, these features such as spectral features 536 can bestored at a step 538 and used as an input to the change-detectionprocess 540.

At a step 542, a co-occurrence textural analysis is performed. This isan example of a combined spatial-spectral pixel analysis. In oneembodiment, a set of spectral co-occurrence matrices are generated forvarious angles using a particular pixel spatial offset. Co-occurrencetexture measures are then computed from these matrices. Multiple pixelspatial offsets may be used to generate different sets of co-occurrencematrices. This textural analysis can be performed over various imagepartitions, such as regions or specific objects in the imagery. Thistexture analysis captures various measures of a region's spectralcharacteristics, as well as the spatial structure of the spectralintensities. Regarding textural analysis, note that a calm pond has ahigher homogeneity texture measure whereas a forest has a high entropymeasure and a low homogeneity measure. We may also compute correlation,skewness, energy, contrast, etc., that can be used to determine texture.Eventually, textural features 546 will be derived which represent thespatial-spectral characteristics of a particular area within theimagery. This area may be a region, an object, or a particular pixel'sneighborhood.

Regarding performing occurrence textural analysis 544, we consider apixel and move a window, passing it over all the pixels in the image.From the pixels under this window we compute occurrence texture measureson a per pixel basis. One of ordinary skill in the art may referoccurrence texture analysis by other names, such as spectral histogramderived texture. These per-pixel measurements characterize the spectralstatistics of each pixel's neighborhood, as defined by the window.Illustrative textural measurements include entropy as measure of therandomness of the spectral intensities within the neighborhood of apixel, as well as skewness of the spectral distribution (measured as thethird order moment of the neighborhood spectral probabilities).Illustrative window sizes include 5×5 and 11×11 pixels for example.These occurrence texture measures could also be included in texturalfeatures 546.

At a step 548 linear features 550 are extracted. Illustrative linearfeatures include length, width, and angle measurements. One method forperforming linear analysis at step 548 includes considering a pixel, andsearching outward in many radial directions (such as 18 directions, likea star burst) and comparing how similar the pixels along a given axisare to the center pixel. Abrupt change in pixel spectral characteristicsis looked for. Where there is little or no change, pixel spectralcorrelation will be higher. Then, a determination is made as to whichradial provided the maximum length of correlated pixels. We also utilizethe minimum length as a width. The length might be one measure, forexample, quantifying whether a pixel belongs to a linear feature. Again,length might be a maximum number of correlated pixels; whereas widthmight be a minimum number correlated. We also derive angularmeasurements of the maximum and minimum lengths a pixel. This captures adescription of such imagery characteristics as roads being at zerodegrees versus ninety degrees. Analysis of the relationship between themaximum and minimum lengths, as well as their angles, providesstructural information about an area of imagery. Illustrative examplesinclude the linear structures found in city road networks (grids)compared to highway interchanges. To clarify, we do not consider thewidth of a pixel to be the correlation length measure at 90 degrees fromthe maximum length. The minimum length may be found at 45 degrees fromthe maximum length, an illustrative example being pixels in the centerof a road intersection.

In one embodiment, a differential morphological profile (DMP) is alsocreated in a step 552. The DMP is an illustrative method that can beused to efficiently perform image segmentation, which basically dividesan image into many segments where each segment or object is comprised ofa group of spatially connected pixels. The DMP uses the morphologicalimage operations opening and/or closing by reconstruction to identifypixel groups that contrast with their surrounding background by usingstructuring elements (or windows) of various sizes. This can be donewith structuring elements of different sizes to determine theapproximate scale of the pixel groups. The DMP is iterativelyconstructed by applying a series of structuring elements and storing ineach level of the profile spatially connected pixels detected at thecurrent scale that were not detected at the previous scale. This will beused in connection with extracting objects, which occurs at a step 554.

An object can be an item within a geographic area or, based on our usageherein, can be a geographic area itself. We will now describe it in thecontext of being an item within a geographic area. When used in thiscontext, an object is a collection of pixels that are deemed to be partof a thing. For example, an object might be a building, or an airplane,or a baseball diamond. Utilizing the results of the DMP 552 is merelyone method for determining objects. Other methods include a host ofsegmentation algorithms to perform object extraction. In one embodiment,each scale of the DMP can be thresholded at some value such that onlypixels with values higher than the threshold remain “on” in a binaryimage. A connected components algorithm can be applied to the resultingimages in order to identify individual objects. After this, eachcomponent in the image can be treated as an individual object.

Turning to FIG. 5C, an illustrative method for extracting objects isprovided. At a step 556, the objects are identified that are within agiven region. These objects are referenced by 558. At a step 560, objectfeatures are extracted. One embodiment of this method includesextracting object-shape attributes at a step 562. One method ofperforming this includes aligning an object on a principal axis and thenrescaling it to a common size. For example, an airplane can bedetermined to have a certain principal axis and then rotated so that ouralgorithms become insensitive to orientation because the objects are allaligned along a common principal axis orientation. One embodiment alsoincludes rescaling the objects to a common size wherein the image isrescaled to a consistent size or some common size. We index the result,which takes the form of a binary rescaled image representation ofobjects extracted from the imagery at different scales.

At a step 564, object texture is measured. This can happen in a mannersimilar to that described in connection with processes 542 and 544.Object spectral aggregation occurs at a step 566. This process is alsosimilar to the process previously mentioned in connection with step 534.

The fruit of extracting object features at step 560 includes objectfeatures, which are referenced by 568, which are then indexed out ofstep 569. Returning to FIG. 5B, we show that object features 570 arestored in step 538, and can also be a source for the change detectionprocess 540. Another feature of objects that can be extracted includefeatures related to an object's shape. Shape features are captured toallow comparison of the shape of an object to other objects in thedatabase. Shape can be represented in numerous ways, an illustrativeexample is to encode the shape as a bitmap. This can be accomplished byreducing the size of an object to fit it into a predetermined referencegrid. Then, grid elements that the shape overlaps are set as on,creating a solid representation. Further details of encoding objectshapes as bitmaps can be found in Chapter 7 of the aforementioned '955Provisional Application (filed on Oct. 16, 2008), which is expresslyincorporated by reference herein. Other illustrative representations mayinclude shape contours or derived skeletons, as well as higher-levelshape measures such as compactness, size, or others.

With continued reference to FIG. 5B, region-based feature extractionoccurs at a step 572, which is shown in greater detail in FIG. 5D. A setof aggregated region vectors 573 is shown in connection with 572.Turning now to FIG. 5D, for a given region, spectral, textural, linear,and object aggregation of pixel-level data occurs. Thus, features thatwere previously gathered in steps such as step 534, 542, 544, 548 and554 (which occur at the pixel level) are aggregated to deriveregion-based features at a step 574. In one embodiment, this aggregationdata is summarized using respective histograms to derive output at astep 576, resulting in a set of region-based features 578. Histogramsare just one illustrative example of numerous possible aggregationtechniques. Other techniques may be employed where appropriate. Anotherillustrative example is that linear pixel-level features are aggregatedinto separate histograms for the angular direction of the maximumrun-length; the ratio of minimum to maximum run-lengths; and also ahistogram capturing the various angles between the maximum run-lengthangle and the minimum run-length angle. In one embodiment, multiplehistograms and some single value aggregates, such as an average maximumrun-length, may be combined into a region-based feature vector 578.These are indexed at a step 580 so that they can be used to facilitateretrieval of information that is sought based on those features. In oneembodiment, feature extraction operates across an entire scene, so thereis no problem of objects or linear structures splitting across tiles.

Briefly returning to FIG. 5B, an output of the feature extractionprocess includes a set of feature vectors 539. In one embodiment, thesefeature vectors describe features associated with each pixel. In anotherembodiment, these feature vectors describe a region or object within theimagery.

Returning to FIG. 5A, we mention that in one embodiment, achange-quality measurement, CQM, is derived at a step 518. This can bethe same CQM as that mentioned in FIG. 2 by reference 240, which we willcontinue to reference. FIGS. 5E-5G depict aspects of deriving CQM 240.At a high level, CQM 240 quantifies a suitability for comparison of theset of comparison scenes. In some embodiments this might take the formof ascertaining the suitability of comparing current remotely sensedimagery to previously stored remotely sensed imagery. If we wereexplaining this in the context of FIG. 2, CQM 240 would quantify asuitability to compare target scene 220 with reference scene 218. Whenwe speak of a suitability for comparing two scenes we are commenting ondetermining a likelihood that any change ultimately detected between twocomparison scenes is the type of change that is sought by someoneseeking change information.

By way of example, factors that tend to affect CQM 240 include asimilarity of times of day that the comparison scenes were taken. Forexample, if two scenes were taken at the same time of the same day, thenaspects such as shadows should be similar across the scenes. Thus,taking into account the times of day that each scene was captured is aninput into deriving CQM 240. Another consideration is the similarity ofsensor-pointing geometries. If a first scene captured by a camera takenat a certain angle, is being compared to a second scene that was takenwith a camera at a similar angle, then such would tend to indicate ahigher suitability that the two images could be compared with eachother. Another data input that goes into deriving CQM 240 might be anyfactor that indicates a likelihood of seasonal differences beingresponsible for image differences. Consider some geographic area.Comparing an image of that area taken during the winter would be lesssuitable for comparing that image to one that was taken during thesummer because snowfall might be a factor. But even if snowfall werepresent, at least if the two images were taken on the same day of theyear, then such might indicate a higher suitability of comparison. Thus,taking into account the Julian day of the year that each image wascaptured is an input into deriving CQM 240.

In addition, latitude could be another factor indicating the likelihoodof seasonal variations because certain geographic areas do not tend toexperience wide seasonal differences. For example, a desert environmentis often always a desert and the Sahara Desert, for example, includesportions that are situated at a certain latitude, and by virtue of thatlatitude, a determination can be made as to the extent that comparingtwo images of a geographic area at that latitude is proper. On the otherhand, a latitude that lies farther away from the equator might generallybe more susceptible to seasonal differences. Therefore, a latitudemeasurement in its own right can be a factor used in determining CQM240.

Sun-aspect geometry can also affect CQM 240. That is, differentgeometries associated with the sun reflecting off a geographic area cantend to produce undesirable change detection results. Sun-aspectgeometry is related to season as well as time of day, so this is yetanother useful factor that can be used in determining CQM 240.

With reference to FIG. 5E, an illustrative process for determining achange-quality measurement (which is referenced by 581 for convenience)is provided. At a step 582, metadata is received from the imagery. Inone embodiment, imagery is received from catalog 314 shown in FIG. 3B.Many of the features discussed are found in the metadata of applicableimagery. For example, metadata 122A includes a latitude indication, atime of year, a time of day, sensor-pointing geometries, and the like.Based on these inputs, at a step 584, constituent CQMs are calculatedincluding a seasonal CQM, a sensor-geometry CQM, a temporal CQM, andothers if applicable. Constituent change-quality measures do not need tobe computed, but are done so in one embodiment. The individual CQMs areaggregated at a step 586 to produce CQM 581, which is associated in 588with some pair of images in a geographic area and stored. Thus, ageographic area might be associated with multiple CQMs. Generally, achange-quality measurement will exist for any two scene pairs thatreflect some common geographic area. This is useful in that it enables auser to search for imagery in dataset 252 that has, is above, or isbelow some given CQM. Thus, in some embodiments this query can be runwith a single constraint; namely, CQM 581.

To recapitulate, in one embodiment, a low CQM value indicates a highprobability that there will be false-positive change detections due toundesirable effects arising from differences in the types of values thatwe have discussed (e.g., time of day, Julian day of year,sensor-pointing geometry, etc.). Illustrative time of day values include0-1440 minutes, Julian day of year can be 1-366 days, illustrativelatitudes can be −90 to 90 degrees, illustrative sensor elevation anglescan be 0 to 90 degrees, and illustrative sensor azimuth angles can be0-360 degrees for example. In one embodiment, a membership function isused for computing normalized (e.g., 0-1) quantities for each componentof the CQM. By way of illustration, an “ESS-function” can be employed,an example of which is shown in FIG. 5F.

FIG. 5G depicts illustrative pseudocode that shows the various itemsdescribed as inputs to a function 590, and illustrative pseudocodevariable definitions 590A. Based on application of the above, thequality of change-detection results decreases as the time separationbetween image acquisition data (delta days) approaches half of a year.This seasonal component depends on latitude where lower latitudes willhave less seasonal variation and a higher quality metric. Of course,there might still be seasonal variations even at the equator, due to,for example, seasonal variations in rainfall patterns (wet vs. dryseasons) or other climatological conditions that cause spectralvariations to occur in vegetated areas.

In an alternative embodiment, a more robust seasonal CQM could bederived from a global analysis of the seasonal variation in the earth'sspectral signature. This data could be derived from satellites employedby agencies such as NASA such as MODIS and LANDSAT.

Regarding the geometry CQM, the quality of change-detection resultsdecreases as the difference between sensor azimuth angles approaches180° (opposite looking directions). This directional difference isrelevant in that undesirable differences between two scenes will resultdue to geometric displacement in the 2D images of 3D objections(buildings, hills, etc.) and their corresponding shadows. The effect ofthe azimuth differences is diminished when the sensor elevation angleapproaches 90°, which is when the sensor is looking straight down,sometimes called the “NADIR” look-at angle.

Regarding the temporal change-quality measure, the quality ofchange-detection results tend to decrease as the difference in imageacquisition time of day (delta time) increases. This will generally havelittle effect on most satellite-based imaging systems because they tendto be sun-synchronous and will pass over the same location on the earthat the same time of each day. However, the temporal CQM metric is usefulwhen dealing with remotely sensed imagery from airborne platforms (e.g.,airplanes, UAVs) which acquire images at all times of the day, includingnighttimes.

Turning now to FIG. 6A, an illustrative block diagram is referencedgenerally by the numeral 600, which will be used in connection withdescribing an illustrative change-detection process, such as thechange-detection process referred to by numeral 244 in FIG. 2. FIG. 6Adepicts ingested imagery 610, which includes new imagery 612 andcorresponding metadata 614 as well as previously stored imagery 616 andcorresponding metadata 618. Provided portions of previously storedimagery intersect new imagery 612, overlapping imagery 620 will bepresent, which will give rise to a whole set of source scenes 622 aswell as target scenes 624 that, when compared, will yield as manycorresponding sets of change-detection information 626. In examples thatwill be described herein, the source scene will be considered the first,or before, scene and the target scene will be considered the afterscene. To help explain a specific illustrative example, we will makereference to a single source and target scene, including source scene628 as well as target scene 630. Source scene 628 is associated with aset of features or feature information 632, and similarly target scene630 is associated with a set of target features 634. By comparing thesetwo scenes we will ultimately arrive at a set of change-detectioninformation 636, which might include the recently describedchange-quality measure 638 as well as a set of feature differences 640,which represent differences in values of features between 634 and 632.Another item that will be produced according to an embodiment of ourtechnology from source scene 628 and target scene 630 is a change map642, which depicts aspects of change between target scene 630 and sourcescene 628. A crude attempt has been made to illustrate a change betweenthe source and target scenes of FIG. 6A by showing the small rectangle644, which appears only in the target scene 630. The change map can besubdivided into a whole set of subparts, which are collectively referredto by numeral 646. Individual subparts, sometimes referred to herein asregions are referenced by numeral 648. We do not mean to imply that allreferences herein to “regions” are to such subparts, but do sometimesuse the term in that way. Other times, it is just some region, with nospecial meaning; (same for “scene” for example). We may also refer toindividual image subparts as “tiles” or “chips”, and, in theseinstances, we do mean that these are equivalent or synonymous with theimage subparts 648.

As will be explained in greater detail, subparts 648 can be individuallyranked according to the closeness by which their attributes match agiven query, which is why subparts 648 are illustratively shown to bebroken out in tiles lined up in a rank order. The subparts 646 do notactually need to be separately stored, but rather might take the form ofa logical creation such that the tiles 648 are not stored separatelyfrom an overall scene, but which are also separately retrievable basedon a corresponding key or any other value that serves to identify eachof the ranked regions 648.

We will now provide an overview of an illustrative method for carryingout a change-detection process with reference to FIG. 6B. We will alsobe referring to other figures to help illustrate concepts associatedwith the higher-level steps of FIG. 6B. Turning now to FIG. 6B, steps649A indicates that source features as well as target features arereceived. Source features are those associated with some sourcegeographic area and target features are associated with a target area.The source area and target area can certainly be the same. Illustrativesource features that might be received include source feature 632 ofFIG. 6A, and illustrative target features include target feature 634also of FIG. 6A.

At a step 649B, featured differences are generated. This method can belengthy to explain so we will postpone explaining it in greater detailas well as the following steps so as to lay a foundation upon which tobuild. Thus, at a step 649C, a pixel change confidence representation isproduced. This has been previously referred to as a change map,including change map 642 of FIG. 6A. But as will be explained, thephrase change map itself might refer to various objects as the case maybe. That is, those of ordinary skill in the art might refer to the samething by different names or different things by the same name. We mightmake distinctions so as to help explain aspects of our technology but donot mean to necessarily implicate differences per say. If the applicablechange-quality measures have not yet been derived, then they are done soat step 649D, for example, showing this step here reinforces ourprevious assertions that the ordering of steps is not meant to implicatea precise order. Although we have previously described how to calculatethe change-quality measure, it might be the case that in someembodiments it is calculated at some different time including during thestep of 649D. Similarly, if the change-map visualization image has notyet been created then it is done so at step 649E, and feature aspectsare aggregated at a step 649F, thereby producing region-aggregationhistograms. This data can all be stored at step 649G, which will bedescribed in later detail below.

Having provided a brief overview of an illustrative change-detectionprocess, we now will turn to FIG. 6C to describe generating featuredifferences (649B) in greater detail. FIG. 6C shows an illustrative setof features, which are referred to by numeral 650A. Numeral 650A showsseveral feature bands, each represented by a row. For example 650B-Kindicate various features that might be associated with the given pixel.Each pixel in a given image can include multiple features and thus,multiple feature values. That is, each pixel might have a red value, ablue value, a panchromatic value, etc. Each of these features isrepresented by an individual row. For example, row 650B describes afirst spectral feature for each pixel in image 650M. Illustrativespectral features include, as mentioned, a red value, a blue value, apanchromatic value, a near-infrared value, and a normalized differencevegetation index (NDVI) value. Each of these features are also referredto as bands. Thus the first spectral feature 650B is also referred toherein as a given band. For example, one might refer to it as a red bandor a blue band.

We generically refer to various bands by identifiers such as “first” and“second” to indicate that different embodiments can have various numbersof these bands or the absence of these bands. By way of further example,row 650E indicates a first textural feature. Illustrative texturalfeatures include textural features associated with a panchromatic band,or a red band or a blue band, etc. Other features stored might includelinear features as well as shape features.

The columns of FIG. 6C each relate to a different pixel. For example,first pixel 651A is one pixel out of image 650M. Similarly, values areshown for second pixel 651B as well as 651C. Ellipses indicate that eachpixel is described by values associated with the different featurebands. An image might have several thousand or million pixels, in whichcase it would have several thousand or million columns.

Turning now to FIG. 6D, we depict an illustrative flow for calculatingchange according to an embodiment of our technology. A pair ofco-registered multi-temporal scenes 652A is made up of a set of targetbands 652B, which are associated with a target scene; and a set ofsource bands 652C, which are associated with a source scene. Sourcebands 652C are described in greater detail with reference to grid 652D,which include columns 652E which, similar to FIG. 6C, each correspond topixels in the source scene; and rows 652G, where each corresponds to agiven feature band. Feature bands 652G are indicated by characters “B1,”“B2,” “B3.” We do not show this level of detail for each of the gridsthat are depicted in FIG. 6D, but do so with reference to sourcefeatures 652C to provide an example.

Clearly, the values shown in the various grids of FIG. 6D (indeed almostevery table herein) are not actual values. No attempt has even been madeto indicate what these values might be in the real world. They aremerely provided for illustrative purposes to indicate at a high levelthat some value is present.

At a step 653A, if desired, a determination is made as to what thecorrect target pixels should be. As previously described, a target imagewill normally go through a transformation process. We are about todescribe comparing pixel-level feature values between two correspondingpixels. But a question arises as to which is the correct target pixelthat should correspond to a sister pixel in a source scene. Imagine twocheckerboards. Now image that each square on the checkerboard representsa pixel. Now imagine that the two checkerboards are pictures of the samething, the same geographic area for example. One might think that thepixel in row one, column one should be compared in the source image tothe pixel in row one column one of the target image. However, it mightbe the case that the identically corresponding pixel is not the actualpixel that should serve as the comparison pixel in the target image.This is the case because co-registration might be imperfect. To theextent that co-registration is imperfect, and pixels might actually be afew pixels off what they should be, we describe an option, actuallyseveral options, of how to go about determining the correct targetpixels.

FIG. 6E introduces an illustrative feature vector. A feature vector is adata array that holds multiple pieces of information that can beassociated with a single pixel in an image or a region containing agroup of pixels in an image. We will describe these in greater detailbelow.

Turning now to FIG. 6G, we depict a reference pixel-level feature vector680A. Numeral 680B represents several pixels in close proximity to asupposed target pixel. We say “supposed target pixel” because the actualvalue or target pixel is not yet known in this embodiment. Referencenumeral 680C refers to a specific column, which identifies the actualidentically co-located pixel. But it might be the case, as previouslymentioned, that the identically co-located pixel 680C is not the properpixel to select values from because of an imperfect co-registration.Therefore, in this embodiment, we employ a proximity-selection processin which the target pixel value of a given band is selected if it's theclosest value to that of the reference pixel feature value. Considerfirst band 680D. It has a reference pixel-level feature value of five,which is represented in cell 680E. In this embodiment, we consider thevalues of several pixels that are located nearby the identicallyco-located pixel 680C. We choose the pixel value that is closest to thereference value, which in this case is five. Therefore, we select thevalue in cell 680F in this case. In fact, the circled values in thetable represent the values that are selected in the differentsituations, giving rise to the difference vector 680G, which merelyrepresents a difference in the various target values and referencevalues.

Turning briefly to FIG. 6F we show that, at a high level, adifference-feature vector 670B can be derived by taking a difference ofa source feature vector 670C and a target feature vector 670D. However,we want to make clear that target feature vector 670D may not actuallybe a single vector; that is, a single column in any of the tables ofFIG. 6G, 6H or 6I. Rather, FIG. 6F is meant to illustrate the concept ofgenerating a difference vector by differencing source and targetpixel-level feature vectors. For example, if the embodiment justdescribed were employed, then target feature vector 670D might becomposed of values 5, 7, 5, 9, 10, etc., based on the circled values ofFIG. 6G.

Turning to FIG. 6H another embodiment for determining the correct targetpixel set is provided. In this example, the identically co-located pixel668A is used to calculate a difference in connection with the referencepixel-level feature vector 668B. This is shown by having each value inthe identically co-located pixel 668A circled. Thus, in this embodiment,difference vector 668C is generated, which represents the differencebetween the circled values and the corresponding values of referencevector 668B. In some embodiments, the absolute values of the differenceare maintained, but in other embodiments the sign is retained in orderto provide certain types of information. The embodiment of FIG. 6H mightbe applicable in the case when co-registration is performed in such amanner as though a very high confidence exists that the images truly arelined up.

In still another embodiment, and with reference to FIG. 6I, weillustrate an example in which although the identically co-located pixel684A is not used, the values of a single pixel are used. We refer tothis single pixel as the actual corresponding pixel 684B. In thisembodiment, a determination is made that pixel 684B is the actual pixelthat corresponds to reference pixel 684C; and thus, it is the one usedto derive change vector 684D. In this embodiment, this determination canbe made by finding a pixel in close proximity that has the best match(i.e. the lowest total change difference) across all bands. There is noneed to select different values in this embodiment because adetermination is made that pixel 684B really does correspond to pixel684C.

Thus, to summarize, the embodiment of FIG. 6G represents a conservativemethodology whereby difference values are conservatively determined,thereby increasing the confidence level that the change that is detectedis a conservative approach as opposed to having many false positives.But if a determination can be made as to what actual pixel should beused then two variations of that embodiment are described in connectionwith FIGS. 6H and 6I. Returning to FIG. 6D it is indicated that targetbands 653B, shown more clearly in grid 653C includes a set of hashedvalues, which indicate corrected target pixel values. While only onecolumn of hashed values is shown in grid 653C, in one embodiment, allpixels in all bands in the target image could be processed in 653A.

These corrected target pixel values might have been generated by any ofthe embodiments we just described. Source bands 652C provide the otherinput into process 654A, which generates the difference features, whichare represented by numeral 654B, which corresponds to grid 654C. Grid654C includes values that reflect a difference between the values ingrid 652D and 653C. In this embodiment, signs are retained, as evidencedby cell 654D, which has a value of “−1.” The difference features 654Bprovides input to step 655A in which a change-confidence image isproduced. In one embodiment, this is accomplished by utilizing theabsolute values of the contents of grid 654C which are also normalizedin one embodiment. This results in a set of bands 655C, whichcorresponds to grid 655D, which includes normalized absolute values ascompared to those in grid 654C.

These normalized values 655C are calculated, in one embodiment, bydetermining a Z-score, or standard score, for each pixel using thestatistics of the pixel values in each band. This is done by subtractingthe band average from each pixel in the band, and then dividing eachpixel in the band by the band's standard deviation. Another way tonormalize the pixel values is to transform each pixel value into a rangeform 0 to 1 using, for example, fuzzy membership functions. There areother ways to normalize these values, but doing so as described aboveenables a per-pixel aggregation across bands to occur at a step 655E.This aggregation across bands can be done in many ways. In oneembodiment, this can be done using a simple summation of the normalizedpixel values produced by step 655B. In another embodiment, this can bedone using a summation process that includes assigning greater weight tocertain bands and lesser weight to others. For example, spectralfeatures in the bands can be analyzed to determine which pixels might beassociated with vegetation. The normalized differences associated withvegetation pixels could be given a lesser weight, in one embodiment,when performing the aggregation across bands so as to minimize theeffect of possible seasonal variations on the change-detection results.The outcome of step 655E is a single vector 655F, which is shown ingreater detail by vector 655G, which represents a compression of thevarious bands in grid 655D into a single band. In one embodiment, thisrepresents compressing 14 bands into a single band. This single bandrepresents one value per pixel. That is, a measure of change intensityper pixel is included in vector 655F. Again, the values are merelyrepresentative. In some embodiments, the actual values might be integeror possibly floating point values with 15 digits of precision.

At a step 655H, an initial change-confidence image is created, resultingin a change-confidence image 656A. The change-confidence image 656A isan image based on the pixel-level values in vector 655F. Vector 655Fincludes values for every pixel such that an image can be derived fromthose values, and that image is at least an initial change-confidenceimage 656A, which might also be referred to as a change map. But, insome embodiments, it might be desirable to refine the initialchange-confidence image 656A by quantizing or making more discrete thechange values derived from vector 655F. In embodiments where this isdesired, such can occur at step 657A, which results in a change map (orrefined confidence image), 658A. We will describe an illustrativerefining process 657A in connection with FIG. 6J after we finishdescribing the rest of this process.

At a step 658B, change measures for the various regions that theprocesses have been acted on are calculated, resulting in a regionchange measure 658C. If all of the processes described in connectionwith FIG. 6D have already been at the region, or tile level, then thisregion change measure 658C would be a single number that would indicatean amount of change associated with a given region such as one of theregions of the set 648 in FIG. 6A. If this process is carried outagainst all of the regions that are the object of a given query, thenthe objects can be ranked based on an amount of change in each region,thereby producing a set of ranked tiles that are ranked according to anintensity of change associated with a given region.

Change map 658A can also provide input into step 658D, in which achange-intensity visualization image 658E is created. This change mapvisualization image 658E might also be referred to as a change map asused herein. This visualization image 658E is often color-coded as wehave previously described earlier in saying that it is somewhat akin toa weather map, which is color-coded based on intensity of change. Itmight be the case that change map 658A is either gray scale or notcolor-coded in some embodiments.

We will now describe an illustrative refinement process with referenceto FIG. 6J, which was initially introduced by way of step 657A. Turningnow to FIG. 6J, we describe what might be thought of as astack-filtering process. Recall what we are trying to do at this point.If desired, in this embodiment, we are refining a change-confidenceimage such as that of 656A. It might be desirable to refine an initialchange-confidence image because that initial image might not be asgranular or as quantized or as discrete as desired. By forcingconfidence image pixel values to be at discrete levels, changeinformation can be more easily observed by a human being in oneembodiment. We are essentially forcing a continuous range of confidencevalues (e.g. floating point) to take on a smaller number of discrete orquantized value levels (e.g. integer). The resulting quantized imagewith a discrete number of levels can be used to more clearly indicatechange. In one embodiment, we iteratively apply various pixel valuethresholds to determine which pixel values can withstand successivepasses, in which those that survive become stacked upon themselves, likecheckers stacked in a column. Then, those columns with the greatestnumber of checkers are associated with certain values and progressivelyshorter columns are associated with still different values such that weare able to quantify and even color code if desired those columns ofvarying height, and in which generally, the higher columns indicate moreintense change, whereas shorter columns indicate less change.

With reference to FIG. 6J, an initial change confidence threshold 660Aprovides an input along with the feature band 655F of FIG. 6D, whichwill be referred to here by numeral 660B. Say we are dealing withinitial change confidence values between zero and one. The initialthreshold might start off as 0.1. Vector 660B includes a single valueper pixel in which each value indicates a pixel's change confidence.

At a step 660C, the threshold 660A is applied to the set of pixel values660B. The outflow of process 660C will be a binary image in which apixel value of 1 represents change above the threshold 660A, and a pixelvalue of 0 represents change not above the threshold. Thus, those pixelsthat survive this threshold will be added ultimately by way of a binaryoperation. At a step 660D, the binary image is dilated, which isaccomplished using an image morphology operation in one embodiment. Sizeconstraints are applied to resulting amorphous objects from the dilationat a step 660E. In one embodiment, the binary image is processed with aconnected components algorithm to discover amorphous objects. Components(amorphous objects) are eliminated from the binary image if they do notmeet a minimum pixel count (size). At a step 660F, a determination ismade as to whether any change pixels are still present after sizeconstraints of 660E. If not, the iterative process ends and proceeds to660K. If there are, the image that was generated as a result of 660E isadded to an accumulator 660H at a step 660G.

Processing advances to a step in which a determination is made as towhether the threshold processing is finished. If so, the process ends ata step 660K resulting in a refined per-pixel change-confidence image660L. But, if the threshold processing is not finished, then threshold660A is increased at a step 660J to a next threshold value. Anillustrative next threshold value might be 0.2, which might indicate ahigher intensity of change. In another embodiment, the thresholds mayincrease non-linearly. An illustrative example of non-linear thresholdsis (0.1, 0.15, 0.175, 0.187, 0.2, 0.21, 0.25, 0.3, 0.4, 0.5, 0.7, 0.8,0.9). Once again, the per-pixel change-confidence of 660B is evaluatedto find pixel values that might survive this test of higher changeintensity. Those that survive form a binary image in which pixel valuesare either present or not present. Those that survive are added to thebinary image and then the binary image is added to the accumulator atstep 660G and the process starts over at 660J again, where the threshold660A is increased yet again. As the threshold is increased, only thosepixel values having progressively higher amounts of change will be ableto survive and thus will operate as checkers that are being stacked ontop of previously surviving checkers. Ultimately, the stack of pixelvalues, or checkers, will be derived for all pixels, and, as mentioned,those pixel values by way of column heights will be mapped to color orgray-scale intensity to ultimately derive the refined change confidenceimage 660L, and/or change-map visualization image 658E. In anotherembodiment, the thresholds may increase non-linearly.

An illustrative tile-aggregation histogram representing a feature vectoris provided in FIG. 6K and referenced by numeral 664. In variousinstances of our technology, vectors are stored as histograms. Thehistogram of FIG. 6K indicates a relationship between a given query 664Ato a result 664B. This comparison is an illustrative example of theregion aggregation histograms 649F in FIG. 6B.

Turning now to FIG. 7A, an illustrative data structure is provided andreferenced generally by the numeral 700, which in one embodiment, isused to store the fruits of the various analysis processes that we havedescribed. Considering the aforementioned '955 Provisional Application(filed on Oct. 16, 2008), the many of the aspects of data structure 700are self-explanatory. Therefore, we reference FIG. 7B, which depicts anillustrative process for storing data, which might include an aspect ofstoring the data as shown by numeral 232 in FIG. 2. Although it might bethe case that storing data 232 does not include these steps because someof the data stored is done so by one or more different storageprocesses, and might even store data that is not stored according to thedata model shown in FIG. 7A. For example, in some embodiments,pixel-level attributes are not stored according to the data model inFIG. 7A and the processes that we are about to describe. Rather, in someembodiments, aggregations of pixel-level values are stored.

With reference to FIG. 7B, statistical analysis is performed on the dataat a step 710. When we refer to “the data,” we refer to a variety ofthings. For example, in one embodiment, the data is a collection ofvectors of floating point or integer measurements (feature vectors),each of which corresponds to a single member stored in a database. Thevectors are multidimensional, adding to the complexity of indexing andstoring the data therein. A great deal of information includinglower-level aspects of the data storage process are described in theaforementioned '955 Provisional Application (filed on Oct. 16, 2008),which is expressly incorporated by reference herein for the purpose of,among other things, describing methodologies and data structuresassociated with various facets of storing and retrieving data accordingto our technology. We will continue to describe some high-level aspectsbelow, but rely on the wealth of information described in theaforementioned '955 Provisional Application to fill in the lower-leveldetails.

As mentioned, the various features might be stored in a database such asimagery database 254 of FIG. 2, which is composed of a collection offeature spaces, each of which represents a set of related measurementsabout the database members. Considering FIG. 7E, for example, ared-texture feature space might be a collection of nine texture measuresfrom the red color spectrum imagery band. In one embodiment, eachfeature space has various properties, such as a descriptive name,dimensionality, a suitable distance metric that enables similarityranking, etc. These properties are stored and described in a singleservice in one embodiment, as shown in FIG. 7A.

Although we have previously alluded to this, there can be variousfeature spaces. We will describe four illustrative feature spaces eachof which can include constituent feature spaces. For example,illustrative feature spaces include texture features, spectral features,linear features, and object-composition features. Regarding the texturefeatures, there can be one feature space for each band includingpanchromatic, red, blue, green, near infrared, as well as theaforementioned vegetation index. By way of illustration, co-occurrencetexture measures can include cluster tendency, contrast, correlation,energy, entropy, and homogeneity, which can be computed over variousspatial extents, such as 2 meters, 10 meters, and the like. These arenot meant to be limiting, but illustrative in nature. In alternativeembodiment, occurrence texture measures, like those previously describedabove, could also be used.

Spectral features might include one feature space for each band asmentioned in connection with the texture analysis. Histograms can benormalized and partitioned according to the spectral distribution of therespected bands across a wide set of scenes in the imagery database.

Regarding linear analysis, linear measurements can be associated withregions and include items such as histograms of a maximum correlationrun-length of pixels (length); the sine of an angle between a regionpixel's minimum and maximum length; and a histogram of minimum andmaximum lengths. Other linear features relate to a directional analysis,in which histograms of maximum run-length directions for a given regionare provided.

Regarding object-composition features, a feature space can exist forobjects lighter than their surrounding content, and a separate featurespace for objects darker than their surrounding content as well. Thesecan be derived from the aforementioned DMP object extraction (See e.g.,FIG. 5B). These can represent an object composition of a region, forexample, answering a query as to whether a region contains a mix ofdifferent size objects, a few large objects, or many small objects.

Regarding the aforementioned statistical analysis of step 710, eachfeature space can be treated independently for statistical analysis. Foreach feature in the feature space, the minimum, maximum and mean can becomputed. Multivariate covariance matrices can be computed across thefeature space. This can be done using standard multivariate statisticalmethods. Alternative embodiments of the statistical analysis may includecorrelation analysis, principal component analysis, etc.

At a step 712, the data is normalized. This can mean normalizing thedata or constraining the data to be within a set of given ranges such asbetween zero and one, or negative one and one, or whatever X, Y. Methodsmight include principle component analysis (PCA); subtract the min anddivide by the range; or Z-score. Other methods can also be employed. Inone embodiment, the resulting normalized vectors may or may not be unitvectors.

At a step 714, the data is clustered. Clustering the data provides classassignments for each member of the database, such as database 254, forindexing methods. This helps facilitate indexing methods, such asentropy balanced statistical K-D trees, to exploit class statistics ofdatabase members that are determined to be similar in feature space. Ina first embodiment, all or part of the data can be clustered accordingto ground-truth labels if available. Ground-truth labels representexpert knowledge or database-content groupings. In a second embodiment,the data can be clustered using an appropriate clustering algorithm thatis suitable for the data population. The clustering may be aspace-partitioning scheme, such as a K-means, or a mode-seekingalgorithm.

At a step 716, the data is indexed. This can include three high-levelsteps, which we will briefly mention and then flesh out below. At a step718, service relations are created from service definitions. At a step720, separate indexes are created for the various feature spaces. At astep 722, the indexes are manipulated if applicable. We will nowdescribe indexing the data in lower-level detail.

Returning to FIG. 7A, service definitions provide a way to createrelations for a given service. They include features, classes, classmembers, node members, leaves, leaf members, and, leaf neighbors forexample. Features might be identified by a unique member Key in oneembodiment. The actual collection of feature-space measurements for eachdatabase member can be stored in the applicable rows. Classes (which canbe identified by a ClassID), store a set of class statistics, such asminimum/maximum bounds, mean, and size.

Class members (identified by ClassID+Key) represents association of eachdatabase member to a given class. A node (identified by NodeID)represents an indexing node that is used for content searchingnavigation. Conceptually, a node represents a partition of thehigh-dimensional feature space. Each node contains information such asthe navigation decision and some node statistics such as size andentropy.

Node members (identified by NodeID+Key) list associations of nodes withthe contained members. Node members are used during indexing inductionand maintenance in one embodiment. A leaf (identified by LeafID) is anindex data leaf, which defines a usually small hyper-rectangle inhigh-dimensional feature space (conceptually). A leaf also can includebasic statistics such as minimum/maximum bounds, mean, entropy, andsize. Leaf members (identified by LeafID+Key) lists associations ofleaves with their contained members. Leaf members are used duringretrieval to generate member rankings against a query feature vector.Leaf members are stored separately from node members in one embodimentso that index maintenance can operate without interference withretrieval rankings and vice versa.

Leaf neighbors (identified by LeafID+NeighborLeafID) represent a featurespace and distance metric driven priority of “neighborship” for eachleaf. During content retrieval, it can be expected that the number ofdesired results might exceed the membership of a single leaf that thesearch initially finds. The “neighbor priority queue” provides anordered list of leaves to continue acquiring results from during acontent-based search. The conceptual effect is that a search intohigh-dimensional space spirals outward in high dimensional feature spacecollecting results based on similarity.

By way of illustration, each feature-space can be represented accordingto our data model as a service. A separate index is created over thedata-member population for each feature-space. The feature-space datacan initially be loaded into a feature table. Classes and class-memberassociations are created, which are the result of ground-truth labelingor clustering as previously described. An entropy balanced statisticalK-D tree (EBS) index is created over the feature-space, resulting in asingle index, which can be materialized from the node relation. Anexample of this is provided in the aforementioned '955 ProvisionalApplication, with reference to its algorithm 1.

Index data leaves are analyzed and the leaf neighbor priority cues arecreated into the leaf neighbor relations. An example of this is providedin the aforementioned '955 Provisional Application, with reference tosection 3.2.3.

As mentioned, online manipulation algorithms enable a manipulation ofthe indexes with minimal downtime and still allow efficient and accuratecontent retrieval. This is discussed more fully in Chapter 5 of theaforementioned '955 Provisional Application. Adding new data isdescribed in connection with the “insert” references. Updating features,as well as data removal (delete) are also described, as is manipulationmonitoring and delayed maintenance.

As previously mentioned our data model enables high scalability andperformance to be realized. The data can be distributed across one ormore relational databases that are located on a network, wherein thedatabases are organized as previously described. The resulting indexingcreates data leaves that are groups of similar content. This allows forhigh scalability of the index as the database itself grows. All or aportion of the index can be materialized in memory of a search-agentprocess.

Search agents can interface with the database to provide a service toclient processes. The agents can be co-located with, or networkdistributed away, from the various databases, e.g., imagery databases254. Search agents can perform parallel database-side “leaf ranking” andagent-side “results set merging.” Lower level details of this aspect canbe provided with reference to algorithms 2 and 3 of the aforementioned'955 Provisional Application.

Turning now to FIG. 7C, we show an illustrative database 724 thatincludes various types of information. Database 724 might actually be aconglomeration of databases and the information shown might actuallycome from various sub datasets. As should be clear from our previousdescription, we employ relational-database technology; thus, it shouldbe clear that the data stored will not actually be stored as flat as isdepicted in FIG. 7C. Additionally, in one embodiment, relationaldatabase technology is not used for source or derived images, insteadcataloging their location on a file system or network, as shown byillustrative example in FIG. 3B. The depiction of the data in FIG. 7C isdone so for illustrative purposes, and is also clearly not exhaustive.That is, more information is stored in database 724. By way ofillustration, a key 726 operates as a unique identifier that identifiesa certain geographic area 728, which might be associated with a firstimage 730 that depicts the geographic area in a first state. If severalimages are associated with a geographic area, then an NTH image 732illustratively indicates that as many images as are available areassociated with geographic area 728. If desired, visualization imagery734 is also stored in connection with geographic area 728, as is one orseveral sets of change quality measures 736 which have been previouslydescribed. Several feature vector indexes can also be stored, each ofwhich provides for content-based retrieval from the aforementionedfeature sets. These indexes can indicate features associated with thedifferent images, image-regions, or image items. We have illustrativelyshown textural indexes 738, spectral indexes 739, as well as linearindexes 740. Because we've run out of space we do not show other indexesbut other indexes such as spatial indexes and the like could also beincluded. Change-detection information 741 is also stored in database724. In some embodiments, change-detection information is stored by wayof a change-aggregation vector, which can be represented by a histogramof aggregated pixel-level change vectors.

The scope of change-detection information 741 can be vast. This can beinformation that describes many aspects of change between a first image,and any number of subsequent images that are associated with geographicarea 728. For example, when the various change feature sets are indexedindividually, as shown by reference 742, changes associated withindividual spectral, textural, or linear feature bands can be emphasizedor de-emphasized. Although we will describe this in greater detailbelow, different change features can be weighted so that results arereturned based on the waiting of inputs. For example, a person might beinterested in things that have changed in a first type of spectral way,with no concern to any textural change. In this case, a given index isgiven priority over other indexes. Again, the change detectioninformation itself, although also shown by a single parallelogram, canactually be an immense amount of information represented by severalindexes, which are not shown so as to not obscure the present inventionand for simplicity's sake.

Turning now to FIG. 7D, an illustrative set of multidimensional featurevector (MFV) is shown and referenced by 743. The format of this canvary. For example, in one embodiment data structure 743 indicates MFVsin columns. The dimensionality of the feature vector is K. This can bethe “k” in an Entropy Balanced Statistical k-d tree. N is the size ofthe database for a given index (e.g., blue band as shown). The blue bandspectral characteristics of each of these objects (Region/Tile in thiscase) are aggregated into a histogram (see FIG. 6 k for example), whichis treated as a MFV as well. Data structure 743 illustratively showsthat different measurements (each represented by a row) can beassociated with each of the different keys, as previously referred to.Each key might refer to a specific geographic area. This format would besimilar for difference vectors representing change detection informationaggregated to a region or tile level.

Turning now to FIG. 7E, a table 744 shows an illustrative set of featurespaces. Each feature space depicted will have the same keys of FIG. 7D743, which correspond to certain geographic regions (such as regionsthat cover a given area such as 250 sq. meter region) or imageryobjects. This allows us to reference the database items across thevarious feature sets with a single key. Accordingly, we can use a bandsuch as the blue-spectral characteristics of a query region to findsimilar regions. Of course, this is just one of sixteen that we use toindex regions in one embodiment. Dealing with all sixteen indexes of thedimensionalities listed in table 744 presents issues associated withhandling the multiple indexes, which we will now describe below inconnection with FIG. 7F.

Turning now to FIG. 7F, a query object is represented by 746. A firstfeature-extraction algorithm 758 can be used to extract features fromobject 746 on-the-fly or to lookup features for the object if it is apreviously processed object. These features can then be used to performa search in a first tree 750. A second feature-extraction algorithm 752is similarly associated with a second tree 754. These aforementionedfeature extraction algorithms produce multidimensional feature vectors(MFV) in two different feature spaces. The MVF of each feature space areindexed separately in index trees 754 and 750. The indexes that aresearched are those previously described.

Table 756 indicates search results that are returned incident to a queryof index 754. “Distance” is used as a term to indicate a similarity or acloseness that the search result is to optimum. In the illustrativeexamples shown, smaller distance scores represent more relevant results.An issue is presented when an object is returned based on a certain setof characteristics it has according to one of the two algorithms (752,758) but is not returned by way of the other. For example, it can beseen that second object 760 and fourth object 762 do not appear in bothresult sets. Therefore, their ultimate rankings, represented by X's 764and 766, respectively, are yet to be determined.

To summarize, our system might allow a user to search using informationextended from multiple data sources (e.g., indexes). Each of thesesources might be indexed using different mechanisms, implemented usingvery different technologies, and even represent different types ofinformation. Results returned by searching these disparate indexes areaggregated in a meaningful fusion.

An example of the aforementioned potential difficulties encounteredduring this process can be seen in the example multi-index aggregationshown in FIG. 7F. The region of interest in the image containing thequery object, in this case an airplane, is first passed to two differentfeature-extraction algorithms 752, 758. These algorithms produce featurevectors which can be used to search indexes containing information intwo different feature spaces, 754 and 750. These features could beextracted on demand or simply referenced in a database if the object hadbeen previously processed. The indexes that are searched are thosepreviously described.

The result from each search is a sorted list of results along withcorresponding distance indications that indicate a distance from thequery object to the result object in a given feature space. The sortedlists are referred to by 756 and 757. But if we attempt to aggregate theresults from these sources into a combined list of results using, forexample sum, as the aggregation function, it is clear that we would beunable to complete this task due to missing information in oneembodiment.

We accommodate for this issue by either retrieving, calculating, orestimating applicable values. For example, we employ an absence penaltymethod (APM) algorithm that over retrieves a list of results from eachsource and computes aggregate scores for objects retrieved in a givenresult list. The aggregate scores are computed by using sum (or averageor max or any other aggregation function) to aggregate the score for agiven result based on the individual scores from each result list.

For example, the score for first object 768 in the list of results fromthe search of index 750 is 0.1. The score for the first object 770 inthe list of results from the search of index 754 is 0.2. The sum ofthese two scores yields an aggregate score of 0.3. However, aggregatinga score for the fourth object, 762, reveals that the object is not foundin the top results from source 750. Thus, we estimate a score for theobject in that result list. Because it didn't show up in the list of topresults from 750, we glean that the fourth object must have a scoregreater than 0.4 (the score of the last result in the list) if we wereto exhaustively enumerate the list of results from searching 750. Thus,the score is approximated as a function of the greatest score in thelist. In one embodiment, the score is multiplied by a penalty factor, P,where P is greater than 1.

Turning now to FIG. 8A, a high-level overview associated with retrievinga set of results based on a query is provided and referenced generallyby the numeral 800. A set of spatial/temporal data 810 as well ascontent data 812 provides an ability to search subject to constraintssuch as spatial/temporal constraints 814 as well as other constraints816. Illustrative queries might include something along the lines ofpresent all regions like X from 2001 to 2003; or present all rectangularobjects like some seed object, or do the same but in New York City, orthe same but only show results near hospitals. These constraints can beused to build a query 818. At a step 820, query 818 is received.Incident to receiving query 818, an initial results lists 822 isreturned at a step 824. These results can be presented at a step 826 ona user interface 828.

If applicable, the initial results list 822 can be revised based on userfeedback or additional user interaction. At step 830, a determination ismade as to whether the results are to be revised. If so, additionalinput is received at a step 832, which results in revising the resultsconsistent with the input received at a step 834. This results in arevised results list 836, which can also be presented on user interface828.

As briefly mentioned, user interface 828 might take the form of a viewerapplication. A viewer application can include an imagery viewer, amapping component, a web browser, a virtual-globe viewer, or anotheruser interface that is adapted to present the fruits of theimagery-results query.

Turning now to FIG. 8B, a diagram is provided that will be used toreference retrieval aspects below. As shown, a query 840 includes one ormore inputs. A first type of input might be temporal inputs 842. Theseinputs would indicate various time spans or dates, such as from such andsuch year to such and such year. Timing can be done at as low a level astiming allows, such as down to the month or day level if the informationis available. Another type of input might include a weighting input 844.This is referenced by showing a set of sliders 843, which can take theform of a user interface component that allows a user to emphasize whattype of features should play what type of role in connection withreturning results associated with query 840.

If a user does not care about linear attributes for example, then thecorresponding slider can be moved to the zero end of the spectrum sothat it plays virtually no weight. Similarly, when a certain type ofchange is sought, sliders can be used to indicate an emphasis to beplaced on perhaps a before scene, or an after scene in connection withchange information (see, e.g., slider 847). Other inputs are referencedby 846. Input 846 indicates that a first set of inputs 848 could includea reference image 848A as well as a target image 848B. In this example,if someone were searching for change information, then reference image848A and target image 848B can be compared so that change informationassociated with the two images can be derived, and then dataset 850 canbe searched for other geographic areas that have changed in a mannersimilar to the way that reference image 848 transitioned to target image848B.

Again, by using sliders such as those at 846, the results returned mightbe those results that change in a similar manner but which started in amanner similar to reference image 848A; or in another search, resultscan be returned that had similar change information but that winded uplooking like target image 848B. Another form of input 846 might includethat of 852, which includes a reference image 852A and changed-detectioninformation 852B. In this example, changed-detection information 852B isprovided as well as a reference image 852A.

Another illustrative type of criteria that serves as input 846 mightinclude those referenced by 854, which include a set ofchanged-detection information 854A as well as an “after” image 854B.

And still another embodiment, input 846 might include only an indicationof a geographic area, referenced by 856. In this example, someone mightrequest any information related to things that have changed in a givengeographic area. Although not shown, still another type of input mightinclude change information only. In this example, the change informationserves as the only input, and dataset 850 is searched for geographicareas that are associated with change similar to the change informationprovided as the only input when making up query 840. This would be anillustrative example of other types of inputs, which is genericallyreferenced by 858. Another example of an input 858 includes achange-intensity indication. That is, results can be searched for basedonly upon a given change intensity. One might search for “all regionsthat have changed by X much.” Query 840 is received by user interface860, which facilitates returning a set of results 862, which can bedepicted in UI 860 or some other user interface. Details regardingdataset 850 are provided in FIG. 7C.

Turning now to FIG. 8C, a high-level flow diagram is provided thatillustrates one embodiment of aspects of searching for results incidentto receiving a query 864. In one embodiment, query 864 takes the form ofa feature vector as previously mentioned. In this embodiment, query 864interacts with index root node 866, in which a given feature “K” iscompared to a threshold value of the node 868. If a determination ismade at a step 870 as to whether the applicable query feature “K” isless than the applicable threshold associated with root node 866. Ifnot, processing advances to a right child node 872, in which acomparison 874 is made between the query feature “K” to the threshold ofnode 872.

If, at step 870, query feature “K” was less than the threshold of node866, then processing advances to left child node 876, in which acomparison is made between the query feature “K” to the threshold of thenode 878. Arrow 877 indicates that the search navigation process 880A isrepeated until the applicable leaf node is reached, which is indicatedby numeral 882. The final navigation into the destination leaf node 882is represented by 880B. Incident to reaching leaf node 882, the resultset is accumulated at a step 884. The flow represented in FIG. 8Crepresents an embodiment of the actions carried out by an illustrativesearch agent, which was mentioned above. We will now describe anillustrative result-set-accumulation process with reference to FIG. 8D.

Turning now to FIG. 8D, an illustrative method for building a result setis provided. A result set 886A (also referenced by the letter “R”) isinitially empty. It is received by block 886B, which indicates a rankingof leaf N members against a given query, such as query 864 of FIG. 8C.This provides an output 886D that is received by process 886E, which inone embodiment includes performing a parallel process for each neighborX. A first process 886F includes a merging of the values shown. Process886G indicates a ranking of leaf X members against the query. A productof this process is result set 886H, which is evaluated for size. Thatis, at a step 886J, a determination is made as to whether the size ofresult set 886H is less than the desired size. If so, processingcontinues with the next neighbor leaf X in step 886E. Otherwise, resultsare returned at a step 886K, which results in a set of returned results886M. A lead N neighbor priority queue 886C (see e.g. FIG. 7A) isassociated with step 886E.

Turning now to FIG. 8E, an illustrative method for ranking lead membersagainst a query is provided. A query 888A is evaluated at adetermination step 888B to determine whether constraints are to beapplied, advancing to one of two steps in block 888C. If constraints areto be applied, then processing advances to step 888E, which illustratesillustrative pseudo code that describes a process in which query 888A isan input. If constraints were not to be applied at step 888B, then thepseudo code associated with block 888D is applied in one embodiment. Theoutcome of either of these processes 888C yields result set 888F in oneembodiment.

Briefly returning to FIG. 8B, results 862 are shown as being provided inconnection with processing query 840. FIG. 8F depicts some illustrativeformats that such a result set might assume.

Turning now to FIG. 8F, a given set of results is referenced by numeral889. A first illustrative format of results set 889 is indicated byreference numeral 890. In this embodiment, what amounts to atwo-dimensional array of populated values is returned. In thisembodiment, a set of keys 890A are associated with a ranking score 890B.Thus, a given key 890C is associated with a given ranking score 890D.Ranking score 890D indicates a degree to which a geographic areaassociated with 890C exhibits characteristics that are of the typesought by a given query. Key 890C can be used to retrieve a variety ofinformation. With brief reference to FIG. 7C, we show that key 890C isusable to retrieve items such as a geographic area, an image thatdepicts the geographic area in a certain state, an image or entire setof images that depict a geographic area in various other states,visualization imagery that can be used to present presentations of thegeographic area at the different states, a change quality measure, otherfeature indexes, as well as change-detection information.

An illustrative query might seek an indication of tiles that haveexhibited certain change characteristics in a given geographic area. Asdescribed, our technology enables results to be returned in the form ofportions of that geographic area that are ranked according to one ormore inputs into the query. If a given query has a result set that canbe ranked by a single score, then returning a data structure of thelikes of that depicted in block 890 can be provided.

In other embodiments, multiple ranking scores can be associated with aset of keys. For example, data structure 892 indicates that a set ofkeys 892A are respectively associated with a first ranking score 892B aswell as a second ranking score 892C. Still other ranking scores couldalso be included, which is indicated by reference numeral 892D. Multipleranking scores can be useful in embodiments where the results returnedmight be a function of multiple variables. For example, consider a querythat seeks geographic regions that have changed in a manner similar to away that another geographic area has changed. In such an example, a usermight be interested in results that are most similar to a changesignature. In such a case, first ranking score 892B might correspond tohow similar target geographic areas have changed in comparison to agiven set of source change-detection information. But a user might alsowish to rank the results based on the after state of a given geographicarea. For example, assume that a user wishes to search for geographicareas that have changed in a way similar to a first geographic area,which has changed from a first state to a second state. Let's assumethat the second state looks like a road, and the first state looks likea forest. Incident to processing a query that describes the user'ssearch, results are returned and first ranking score 892B indicatesgeographic areas associated with a change pattern that is similar tothat of the source change signature. But a user might be most interestedin results that have a similar change pattern (e.g., a similar changemap or change intensity) but that wind up looking like the sourcegeographic area; that is, like the image of the source geographic areain the second state. Thus, the user is interested in receivingindications of geographic areas that have changed in a certain way andthat wind up including pictures of a road. If this is the case, thensecond ranking score 892C might indicate a degree of similarity betweentwo after images.

Similarly, if a user was more interested in the results that exhibitsimilar change features but that also appear the way the sourcegeographic area started off; that is, the way it was depicted in itsfirst state. In such an example, second ranking score 892C might be usedto indicate a similarity of two common first states. We have providedtwo illustrative examples to illustrate the concept that search resultsmight be ranked based on more than one score. Sometimes, results 889 areto be revised. For example, a user might provide feedback that indicatesa desire to have an initial result set revised.

Turning now to FIG. 8G, we depict an illustrative process 894 forrevising an initial set of results. Further below in this disclosure, wewill provide examples of how a user could provide user-feedbackindications 894A. Briefly, when query results are presented to a user,they can be presented in connection with a set of user-feedback optionsthat enable a user to provide feedback based on the individual resultsreturned. This can be seen, for example, in FIG. 18. In otherembodiments, pop-up windows can be used to expose a set of links thatprovide additional input that is associated with a set of query resultsalready returned. An illustrative pop-up window or control includes thatof 2412 in FIG. 24. Both of these features will be described in greaterdetail below, but we wanted to briefly introduce them here.

Incident to receiving a set of user-feedback indications, positiveand/or negative tile samples are derived at a step 894B. A tile ranking894C associated with positive tile samples is generated. Also, a tileexclusion list 894D is generated, which is usable to account fornegative tile samples. At a step 894E, the tiles in exclusion list 894Dare removed, resulting in a revised set of rankings 894F. The revisedrankings correspond to re-ranked geographic areas, which are associatedwith corresponding imagery, which is why we also indicate that therevised rankings amount to a second set of images. In some embodiments,the second set of images is a subset of the first set of images, but inother embodiments the regions that are returned are outside of theinitial set. For example, perhaps a user performed an initial search andthen sought a revised set of results based on one or more regions thatexhibited desirable features. Incident to receiving an indication ofthose user-feedback options, an embodiment of our technology wouldenable, for example, searching for other geographic areas that exhibitfeatures that a user indicated as desirable in connection with thechosen initial set of results. In another example, perhaps a user wasinitially searching geographic areas in Missouri, and then sought arevised set of results based on one or more regions that exhibiteddesirable features. Incident to receiving an indication of thoseuser-feedback options, an embodiment of our technology would enablesearching the entire country, for example, for other geographic areasthat exhibit features that a user indicated as desirable in connectionwith the chosen initial set of results.

Turning now to FIG. 8H, an illustrative process for deriving a tilerankings 894C is provided and referenced generally by the numeral 896.An initial positive sample tile (P) 896A serves as input to threeprocesses, which are indicated by reference numerals 896E, 896K, and896P. Starting with the left portion of the flow, atile-feature-intersection histogram 896B serves as a second input intostep 896E, which includes calculating a distance of tiles from sampletile 896A in each feature space using the tile-feature histograms 896B.Again, when we speak of “distance” in this context, we are generallyreferring to a quantified value that indicates a relevancy to a providedquery. Step 896E is done as many times as histogram inputs are provided.For example, one embodiment of our technology utilizes 14 tile features,and thus 14 histograms. In such an embodiment, step 896E is performed 14times. The tile distances are aggregated across features at a step 896F.At a step 896G, the tiles are ranked according to the aggregate tiledistance of step 896F. In one embodiment, the ranked order is ascending.The fruit of this step is an initial tile ranking 896H, which indicatesfeature distance. This tile ranking 896H is inputted into step 896J,which calculates the sum of order statistics (SOS) from all the rankingsfor each tile. While not shown in FIG. 8H due to space limitations,feature histograms associated with the “before” tiles and featurehistograms associated with the “after” tiles could follow the sameprocessing sequence associated with the tile feature intersectionhistograms 896B. In one embodiment, this would allow for feedbackassociated with the “before” and “after” tile states to return rankedresults that are more like one or the other state. We will describe howthe other tile rankings are obtained now.

Returning to step 896K, which contemplates calculating a distance oftiles from sample tile 896A using the tile-confidence histogram 896C. Ata step 896M, the tiles are ranked based on their respective distancescores. In one embodiment, this ranking is in ascending order. An outputof this process is a tile ranking 896N, which relates to confidencedistance. This tile ranking 896N is also inputted into step 896J in oneembodiment.

Returning to step 896P, which contemplates calculating an absolutedifference of tile-aggregate confidence, inputs include positive sampletile 896A as well as an indication of tile-aggregate confidence 896D.From this, the tiles can be ranked using absolute tile confidencedifference, thereby yielding a tile ranking 896R, which relates toaggregate confidence difference 896Q. This ranking 896R is also inputtedinto 896J, which yields an indication of a set of tiles sum of orderstatistics 896S, which is relevant to sample tile 896A. This process canoccur for multiple positive sample tiles. For example, a user mightindicate multiple tiles as being the type of tiles of interest to him orher. Thus, all of the applicable rankings from multiple positive sampletiles are received at step 896T. The tiles can be ranked using a totalSOS value at a step 896U, which results in a tile ranking 896V that isrelevant to the positive tile samples. This tile ranking 896V can be thesame as that of 894C in FIG. 8G.

Turning now to FIG. 8I, an illustrative method for constructing atile-exclusion list 898D is provided and referenced generally by thenumeral 898. A first input includes a negative sample tile (P) 898A. Anegative sample tile could be a tile that a user indicates isundesirous; that is, as having features that are undesirable to a user.Other inputs include a set of tile-feature intersection histograms 898B,a tile-confidence histogram 898E, as well as a tile-aggregate confidence898M.

Tile-feature-intersection histogram 898B is received along with sampletile 898A at a step 898C, in which a calculation is made of the distancethat the tiles are from sample tile 898A in each feature space using thetile-feature histograms 898B. These distances can be aggregated acrossthe various features at a step 898D, after which processing advances toa step 898G in which calculation is made of the closeness of each tileto sample tile 898A.

Tile-confidence histogram 898E is inputted into step 898F along withnegative sample tile 898A so that the distance of the tiles from sampletile 898A can be calculated. Processing advances to step 898G, in whichthe closeness that each tile is to the sample is calculated. Similarly,at a step 898N the absolute difference of tile-aggregate confidence 898Mis calculated based on negative sample tile 898A. The outputs of thisstep 898N are also inputted into step 898G, after which processingadvances to a step 898H in which the tile is excluded if its degree ofcloseness is too high. Here, we are looking to exclude tiles similar tothe provided negative sample 898A.

At a step 898J, tiles are included in the tile-exclusion list, as areother tiles that have been acted on in addition to sample tile 898A.This results in creating tile-exclusion list 898K, which can be the sameas tile-exclusion list 894D of FIG. 8G.

We will now summarize in some respects and in other respects provideadditional examples of aspects of our technology. From this pointforward, although we might make reference to other figures as well, wewill describe some examples in connection with FIG. 1B, which depicts afirst geographic area 110 as well as a second geographic area 112. Eachof the geographic areas are associated with imagery taken at least twodifferent times. Clearly, if images were taken at multiple times, thenmultiple states would be depicted. But as shown, each area is associatedwith a first state and a second state. First geographic area 110 isassociated with an image that depicts a first state 114 as well as animage that depicts a second state 116. Various features 118 areassociated with a geographic area imagery of the first state, andcorresponding features 120 are associated with the image that depictsthe second state 116 of first geographic area 110. A correspondingchange map 122 indicates change associated with change from first state114 to second state 116. Similarly, a change quality measure 124 as wellas a set of difference-feature vectors 126 are also associated withchange between first state 114 and second state 116. We do not mean toimply that these are necessarily associated with change map 122, butschematically show them in close proximity to change map 122 to conveythat they convey change information associated with first geographicarea 110. Similarly, second geographic area 112 includes imagery 130that depicts it in a first state as well as imagery 132 that depicts thesame area 112 in a second state. First state 130 is associated with aset of features 134, and second state 132 is associated with acorresponding set of features 136. A change map associated with firststate 130 and second state 132 is referenced by the numeral 138, as wellas a corresponding change quality measure 140 and set of change features142. As with the first geographic area, second geographic area 112 hasimagery taken at two times, thereby providing first state 130 and secondstate 132. In other embodiments, multiple images taken at multiple timescould be employed to derive any number of states of the two geographicareas in overlapping portion 150, which intersects the two geographicareas.

Turning now to FIG. 9A, an illustrative method for detecting change in ageographic area is provided and referenced generally by the numeral 900.At a step 902, a set of remotely sensed imagery is received that depictsthe geographic area. For example, remotely sensed imagery 152 might beapplicable.

At a step 904, changes in physical features of the geographic area areautomatically identified. We have previously described illustrativeembodiments of change-detection processes that would identify changes inphysical features between two states of the given geographic area. Forexample, if one were considering second geographic area 112, this stepwould include detecting changes between first state 130 and second state132.

At a step 906, a change-quality measurement associated with the set ofremotely sensed imagery is derived. We illustratively showchange-quality measurement 140 in connection with second geographic area112, which is associated with imagery that depicts first state 130 aswell as second state 132. We have also discussed methods for calculatingCQM 140 above. See, for example, FIGS. 5E-5G. As mentioned, thechange-quality measurement 140 quantifies a suitability of comparison ofthe set of remotely sensed imagery to previously stored imagery. Thus,first state 130 is associated with previously stored imagery, and, byway of example, second state 132 is associated with newly acquiredimagery. CQM 140 indicates a suitability for comparing the two images.We have previously mentioned that illustrative attributes that factorinto determining CQM 140 include a similarity of times of day, asimilarity of sensor-pointing geometries, as well as any factor thatindicates a likelihood of seasonal differences being responsible forimage differences. Such a factor might include an indication of asimilarity of days of year, a similarity of sun-aspect geometries and agiven latitude at which the images were taken.

Turning now to FIG. 9B, a more detailed flow diagram that illustrates anillustrative method of detecting change in a given geographic area isprovided. The change that could be detected does not need to include anentire area but might include portions thereof or even items therein. Ata step 912, the initial imagery is received that includes images andmetadata. Illustrative initial imagery could include that of FIG. 1Aillustrated by numeral 118A. In such an embodiment, modified imagery isderived from the initial imagery at a step 914. In other embodiments,what is referred to as modified imagery might not need to be derived ifit is already corrected in a manner that makes it suitable forcomparison with other images.

At a step 916 the modified imagery is stored. In one embodiment, thestored modified imagery has been subjected to a geometric-correctionprocess prior to being stored, thereby compensating for imagerydistortions due to terrestrial topographic features. We have discussedthis above. Storing the modified imagery might also include resamplingthe initial imagery to a nominal resolution, such as a common resolutionshared between comparison images. In still other embodiments, themodified imagery includes initial imagery that has been geometricallycorrected, resampled to a nominal resolution, had its pixel valuesconverted to reflectance, and reprojected onto a common coordinatesystem. Illustrative coordinate systems include UTM, state plane,geographic, military-grid, etc., as previously mentioned.

At a step 918, overlapping imagery is identified. This overlappingimagery can include two comparison scenes, such as a target scene and areference scene. In one embodiment, the target scene includes a portionof modified imagery that describes a certain geographic region, such asfirst geographic area 110 in FIG. 1B. Multiple reference scenes can besubsets of formerly stored imagery that also describe the samegeographic area, such as first geographic area 110. Thus, in thisexample, first state 114 might take the form of a reference scene andsecond state 116 corresponds to a target scene. As mentioned,identifying overlapping imagery can be accomplished by coregistering thetarget scenes with each other, or coregistering the target scenes with asingle reference scene that serves as a common basis for coregistrationof multiple scenes (see, e.g., FIG. 4B).

At a step 922, changes are automatically detected without userintervention between however many scenes are desired, such as in betweenfirst state 114 and second state 116. In one embodiment, automaticallydetecting changes include referencing change-quality measure 124. Insome embodiments, merely querying on change-quality measure 124 canprovide a certain set of desired results.

In one embodiment, automatically detecting changes includesautomatically extracting a set of target features 120 from target scene116, comparing the set of target features 120 to stored referencefeatures 118 associated with reference scene 114. Automaticallydetecting changes can also include producing multiple sets of featuredifferences 126. We show 126 as including multiple sets wherein each setmight be associated with a sub portion of first geographic area 110.This results in a group of feature differences 126. As mentioned, thefeatures might include spectral features, spatial features, linearfeatures, or textural features. These, of course, are not the onlyfeatures that might form a part of feature set 118 or 120, but areprovided by way of illustration. Those of ordinary skill in the artmight associate different wording with our terminology. For example, twomain features include spectral and spatial features. Some features are acombination of these, such as wavelet, linear, and textural beingderived therefrom. We do not mean to make any limiting comments inconnection with explaining illustrative features and types of featuresthat could be extracted and form parts of sets 118 or 120.

Automatically detecting changes (step 922) can also include deriving achange-map 122, which indicates areas of change between the comparisonscenes 114 and 116. As we hopefully made clear previously, change map122 depicts much more than whether there was change or not, especiallyat the pixel level. For example, one approach might be to simplyindicate whether a change is associated with a given pixel, or changeabove some threshold. We do much more than this. We indicate intensityof change as well as area of change, and will show examples of this inother figures (such as FIG. 15, referencing numerals 1518 and 1520). Aspreviously described, the areas of change identified in change map 122can be gleaned by subjecting pixels to multiple successive thresholdsthat ultimately make up the change map and that also indicate pixel-wisechange; that is, change at the pixel level.

In some embodiments, the change-detection or other process furtherincludes subdividing change map 122 into a set of subparts, determininga change score associated with each of the subparts by aggregatingpixel-wise difference values between pixels within the subparts, andranking each subpart based on the change score, which indicates anamount or intensity of change from first state 118 to second state 120.

At a step 924, the sets of change-detection information 126 are storedsuch that they are available to satisfy query-seeking informationrelated to it. We have described manners in which the change-detectioninformation can be stored so as to facilitate efficient retrieval aswell as efficient storage. This enables the change-detection information126 to be added to a set such as dataset 252 at FIG. 2 so that it can beacted on by a query seeking specific results.

Step 926 describes deriving a change map that depicts intensities ofchange over regions such as first geographic region 110. At a step 928,a query requesting information associated with a given region can bereceived. An initial set of results can be provided at a step 930, whichcould be presented on a user interface. In some embodiments, user inputis received at a step 932 requesting a revision of the initial set ofresults. When we speak of a revision of the initial set of results we donot mean to imply that the second set of results that are provided (step934) all come from the first set of results. To the contrary, it mightbe the case that all or a portion of the second set of results were notpart of the initial set of results returned.

As briefly mentioned, the results can be presented on a presentationdevice so that the change information can be visualized. We will provideexamples of such visualization below. One aspect of visualizationincludes presenting the aspects of change (such as those described bynumeral 126) in context such that the aspects of change are presentedamong surroundings that have not changed beyond some amount. As we willexplain, tiles can be ranked and then pushed out to an image viewer suchthat only small chips are highlighted, but are done so among abackground, which enables the chips to be viewed in context; that is, inone embodiment, in the midst of surrounding geographic areas. In oneembodiment, this push can occur through the delivery of a KML documentthat contains references to the imagery to be displayed.

Turning now to FIG. 9C, another embodiment of a method for detectingchange in a geographic area is provided and referenced generally by thenumeral 940. Provided steps similar to that of FIGS. 9A/9B wereaccomplished at some point, at a step 942, change-detection informationis automatically derived that is based on differences between a firstfeature set of first imagery and a second features set associated withsecond imagery. For example, change-detection information can be basedon differences between features set 118 and features set 120.

At a step 944, portions of changed-detection information 126 arelogically associated with corresponding portions of a geographic areasuch as first geographic area 110. That is, in one embodiment, firstgeographic area 110 might be subdivided into a set of subareas. Weillustratively showed a set of subareas on FIG. 6A (see, for exampleitem 648). In one embodiment, sub-portions 648 are not actually createdor separately stored, but are logically defined to facilitate a logicalassociation of an individual geographic sub-portion with a set ofchange-detection information. But in other embodiments, the regionsmight be actually stored depending on implementation preferences.

At a step 946, the change-detection information is stored in a datasetsuch that the regions of the geographic area can be retrieved on demandto satisfy a query seeking change information within the geographicarea. Portions of the change-detection information respectivelycorrespond to regions of the geographic area.

At a step 948, a query is received that has certain inputs. The types ofinputs can vary. Illustrative inputs include an indication of a givenarea and a time indication that indicates a time span over which changeinformation is sought. Thus, in one embodiment, the inputs to a queryinclude an indication of the area over which change information issought as well as a time span. For example, a user might be interestedin receiving change-detection information associated with the City ofSpringfield from 2001 through 2007. Other information could be providedas user inputs as well. For example, two images might be provided,wherein change information is gathered based on the two images, and thechange information is used to search for other geographic areas thathave changed in a manner similar to the change information derived. Aspreviously mentioned, the inputs might also be only a single input, suchas only a region indication or only a change-intensity indication. Inthe case of only a region indication, any change information associatedwith that region indicated would be provided in response to such aquery. Similarly, if only change-intensity information were provided asa query input, then only those geographic areas that satisfy thechange-intensity-information constraint would be returned. Many moreoptions exist, and we will provide even more examples below.

The process 940 could also include identifying a set of target regionsof the geographic area based on the inputs. The geographic area could beassociated with a before image that depicts a given target region in afirst state and an after image that depicts the given target region in asecond state. By way of example, if the target region is firstgeographic area 110, then a before image might be the image associatedwith first state 114 while the aforementioned after image might be theimage associated with second state 116.

At a step 950, results can be returned based on the query. In oneembodiment, the results take the form of a ranked list of targetregions, or identifiers capable of identifying the target regions.Attributes that have changed that can be searched on include a changeintensity, a change-quality measure, a change in spectral features, achange in textural features, a change in linear features, and a changein shape features.

We will now describe illustrative methods for identifying terrestrialobjects that change in a certain manner. We will reference FIG. 8B butturn now to FIG. 10A. FIG. 10A depicts such an illustrative process,which is referenced by numeral 1000. At a step 1001, a query is receivedthat includes inputs that indicate attributes of terrestrial objects ofinterest. With reference to FIG. 8B, we show just a sampling ofillustrative inputs to a query. For example, by way of reference numeral846, a first set of inputs might include a reference image and a targetimage, 848. A second set of inputs 852 might include a reference imageand change-detection information that is associated with that referenceimage. Similarly, as mentioned, another illustrative one or more inputsmight include change-detection information 854A as well as a targetimage 854B. A geographic area indication 856 might be provided. A CQMvalue might also be provided. Temporal information 842 might beprovided. Weightings can also be provided by way of a control such asthat of 843. These weightings can be used to emphasize or de-emphasizevarious types of change features. In one embodiment, these weightingswill be supplied to a result-set fusion process as shown in FIG. 7F.Similar sliders could also be used to provide emphasis given to thereference image 848A versus the target image 848B such that results lookmore like one or the other in connection with correspondingchange-detection information.

At a step 1002, a source set of change-detection information isdetermined from the inputs of the query. For example, consider inputs848 in FIG. 8B. Change detection might be gathered by referencing thegeographic area associated with each of the images, and thencorresponding change-detection information, which for example, might bechanged-detection information 851. In another embodiment, ifchange-detection information was provided as an input to the query, thenit will simply be referenced. An example would be providing CDI 852B orCDI 854A. In still another example, if a geographic-area indication 856was provided, then its corresponding key from dataset 850 can bereferenced to locate pre-stored change-detection information.

In still other embodiments change-detection information can bedetermined on the fly. That is, if images 848 were provided,change-detection information can be derived from those two images andthen used in connection with seeking results from the applicabledataset.

The results of the query will be identified images that were derivedfrom high resolution, remotely sensed imagery and that are associatedwith certain terrestrial objects (which might be geographic areasthemselves) that have changed in the manner similar to a certain manner.

At a step 1003, a dataset is referenced that stores sets ofchange-detection information that are respectively associated withvarious objects. We show an illustrative dataset in FIG. 8B, referencedby numeral 850; another in FIG. 2, 252.

At a step 1004, results from the dataset are determined based on thesource change-detection information. In one embodiment, the resultsinclude a set of images that includes content that is associated withchange-detection information that is similar to the sourcechange-detection information.

At a step 1005 the images are ranked based on their similarity to thesource change-detection information. An indication of the results can bepresented based on the ranking values at a step 1006.

Turning now to FIG. 10B, another illustrative method for identifyingterrestrial objects that have changed in a certain manner is providedand referenced generally by the numeral 1008. At a step 1010, a query isreceived that includes inputs that indicate attributes of terrestrialobjects of interest. That is, a query is received that indicatesattributes that would be possessed by objects that a user wishes tolocate.

At a step 1012, a source set of change-detection information isdetermined that is associated with a given source object. We havepreviously described aspects similar to this step in describing step1002 in FIG. 10A.

At a step 1014, a dataset is referenced that stores sets ofchange-detection information that is respectively associated withvarious objects.

At a step 1016, results from the dataset are determined. The resultsinclude, in one embodiment, images that include content that hasfeatures similar to those of the source object and that are associatedwith change-detection information that are similar to a set of sourcechange-detection information. In one embodiment, determining the resultsincludes identifying a source feature set that describes a set offeatures associated with a first image, comparing that source featureset against stored featured sets within the dataset, and determiningwhich of the plurality of stored featured sets most closely matches thesource feature set.

Consider, for example, FIG. 1B. Identifying the source feature set mightinclude identifying feature set 118. Then, the dataset might be searchedfor other images that share aspects of feature set 118. For example,items such as first state 130 might be retrieved if the dataset issimilar. In other embodiments, feature set 120 might be used as a sourceto search upon. Still further, change-detection information 126 might beused to locate other geographic areas. For example, perhapschange-detection information 142 is similar to change-detectioninformation 126. If so, then second geographic area 112 might bereturned in response to a query based on first geographic area 110.Thus, determining the results includes determining target objects thatare similar to the search object in the manner just described.

At a step 1018, the images are ranked based on their similarity to thesource object and to the source change-detection information. Aspreviously explained (for example with reference to FIG. 8F), the rankedresults might be based on one or more ranking scores.

At a step 1020, an indication of at least a portion of the results arepresented based on the ranking.

Turning now to FIG. 10C for another illustrative method for identifyingterrestrial objects at a change in a certain manner is provided andreferenced generally by the numeral 1030. At a step 1032, a query isreceived that includes input that is related to a first terrestrialobject and to source change-detection information that describes changeassociated with the first object. Illustrative objects includegeographic areas or items within the geographic areas. The sourcechange-detection information is the change-detection information that isused as a source upon which the query utilizes to search a dataset.

At a step 1034, the query is applied to a dataset that includes indexedinformation that describes remotely sensed imagery. As mentioned, theindexed information can be separately indexed based on individualfeature sets.

At a step 1036, an initial query result is received based on the inputs.The query was satisfied in one embodiment by identifying the set ofregions in the dataset that are respectively associated with changeinformation that is similar to that of the source change-detectioninformation. The regions can be ranked at a step 1040 in one embodiment,and then indications of the identified regions can be presented at astep 1042. In one embodiment indications include textual references tothe various geographic regions that are returned, and in otherembodiments pictorial information is displayed.

If applicable, the initial set of results presented can be revised at astep 1044 incident to receiving user feedback, and then presentedafresh.

We will now describe in greater detail aspects associated withvisualizing results returned from a query. Various types of queries canbe applied against datasets 252. A first illustrative type of querycould be a region-based query, in which a user is seeking geographicareas of interest based on content provided. Images can serve as thebasis for query inputs. Inputs can be selected by way of indicating apre-stored image, or on-the-fly-type of embodiments can also beemployed. For example, a user might hand draw or utilize acomputer-aided-drafting program to outline a shape of an item ofinterest. For example, the user might be interested in locating thingsthat appear in a shape similar to an airplane. The user could draw anairplane, in which this drawing serves as input into a query. The imagesshape features are analyzed and used to search against the data indataset 252.

Input could also take the form of grabbing a tile or data from anotherapplication. For example, an earth viewer might be depicting a certainarea of land. That area of land can be grabbed and inputted into thesystem as one of potentially multiple inputs that make up a query. Ifcorresponding metadata is not available, then a search might be limitedin some respects. In still other embodiments, imagery or items can beuploaded to serve as the basis for a query.

Returning to an explanation of the various types of inputs that mightmake up a query, geospatial information could be provided. We do notmean to attach a term-of-art type of definition to this term.Illustrative location-type information might take the form of “inMissouri,” or “within ten kilometers of X.” X could be a wide variety oftypes of items. For example, X might relate to types of organizations.Illustrative types of organizations include things like schools,churches, hospitals, energy-generation facilities, and many more. Theexamples are legion. X could also take the form of geographic features.That is features that exhibit certain geographic characteristics such aslakes, other bodies of water, canyons, and the like. X could also relateto transportation features, such as harbors, airports, etc. X mightrelate to populated areas, such that a query might request geographicareas that are within some proximity to a given populated area. X mightrelate to military establishments, such as airbases, hospitals, and thelike. It should be clear that the few examples we gave are illustrativein nature. We have previously mentioned that these heterogeneous typesof data items can be levied as constraints that form part of a query.

Another illustrative input could include a temporal input. Anillustrative temporal input might be on the order of “between time A andtime B,” or “anything after to such and such date.” Any constraintrelated to time can serve as an input into a query.

Another input to a query might be an indication of object size. Forexample, a query might desire to be constrained to those objects “withina size between 30 and 80 square meters.” Item size itself can alsooperate as a constraint, such that buildings that consume some minimumor maximum square area are searched, or items occupying a given amountof space.

As previously mentioned a change-quality measure can be an input. Anillustrative search involving a CQM might be something along the linesof “with CQM greater than 8”. This can help filter out false positivesand reduce a set to include only those results that might be morerelevant to a user.

As mentioned, a change intensity itself can provide the basis for aquery input. In one embodiment, a provided change intensity can operateas a threshold. An illustrative search involving change intensity mightbe something along the lines of “find all regions or tiles in KansasCity, Mo. with a change intensity greater than 5”. This can help filterout regions with low, weak, or no change, thereby reducing the resultset to include only regions that have strong change which are morerelevant to a user.

Of course a query might include any combination of the above inputs aswell as other inputs not mentioned. This allows for sophisticated querysearches, such as “return all change tiles above some threshold withintwo kilometers of all airfields in California that are within threekilometers of a park within the last six months that have achange-quality measure of at least 7”. Query inputs can also begenerated from results that were returned from a previous query. Forexample, in the aforementioned query, if a given set of results arereturned, the user might wish to find more results that exhibit featuresthat are similar to one or more particular results in (or not in) theresults set. In this case, the results of a former query can becomeinputs to a subsequent query.

As briefly mentioned above, the different inputs to a query can also beweighted. That is, varying degrees of importance can be associated withthe different attributes so as to fine-tune the set of results that arereturned incident to submitting a given query. We will describe this ineven greater detail in connection with some illustrative user-interfacediagrams that are discussed below.

Sometimes, the database dataset-searching approach is dictated by thetype of input provided. For example, if the input is a given pair ofscenes, then content associated with the scene such as feature vectorsor change-detection information can be referenced and then used tolocate items of interest. By way of illustration, inputs might include afirst scene, a second scene and a change (difference) feature vector.

An on-demand embodiment contemplates receiving two new images that havenot been ingested. In such an embodiment, feature attributes associatedwith the images can be derived on demand, and those feature sets can beused to search against data in dataset 252.

We briefly mentioned that region-based queries are one example of thetype of query. Other types of queries include item-based queries, whichsearch for items within regions, as well as change-related queries,which either search for change or search for geographic areas that havechange in a manner similar to the way that other geographic regions havechanged.

FIG. 11A depicts an illustrative presentation 1100 that shows variousways of receiving geographic user inputs to facilitate a query. By wayof example, it is shown that items such as latitude, longitude,proximity to a given area, a country, a classification of a type ofitem, and more are shown. In some embodiments, prior searches 1110 canbe presented to a user. The prior searches 1110 provide otherillustrative examples of types of searches that are applicable to ourtechnology.

FIG. 11B depicts a screenshot 1114 that illustrates the sliders 1116that we have previously mentioned. As shown, a first slider 1118 isassociated with spectral features, a second slider, 1120 is associatedwith texture features, a third slider 1122 is associated with linearfeatures and a fourth slider, 1124 is associated with objects orscale-related features. We do not mean to imply that sliders are theonly way to vary a weighting of inputs into a query. Textboxes couldalso be used, which are actually also shown. One of the points to betaken away from user-interface component 1116 is the fact that variousinputs can be weighted, and then results from the various feature spaceindexes are weighted during the aggregation process of result-setmerging as discussed previously and illustratively depicted in FIG. 7C.In one embodiment, the result objects' distances are multiplied by theweight of the feature space during aggregation. One way of accomplishingthis is to store attributes of specific features when storing theimagery. For example, when we store an image associated with a givengeographic area, we also store attributes at the feature level. In somecases, this results in storing information associated with multiplebands, such as 14 bands or 16 bands of information. We have providedillustrative types of bands or features in, for example, FIG. 7E, whichindicates various features that can be separately indexed, and thusseparately searched on, and thus separately weighted so that the resultsreturned are those that are relevant to a user based on a weightingprovided. Screenshot 1114 indicates that a given image 1128 provides thebasis for a query, which in one embodiment is selected from a group ofpre-stored example tiles 1130.

FIG. 11C depicts a screenshot 1132 that provides another example ofquerying based on an image, such as image 1134. This embodimentindicates that a result 1136 could be returned and also shown incontext. Other results include items 1138 and 1140.

FIG. 11D includes an item 1142, which illustrates the fact that anentire set of result tiles can be presented to a user in a way, such asin a grid-like pattern. The tiles shown could also be pushed out to animagery viewer, which we will describe in greater detail below.

Turning now to FIG. 12A, a state diagram depicts an illustrative methodfor updating one or more viewers based on input provided. An initial setof input 1210 might be received by first viewer 1212 in one embodiment.First viewer 1212, is coupled to a server 1214, which is coupled to asecond viewer 1216. We indicated by ellipsis 1217 that many more viewerscould also be utilized in connection with an embodiment of ourtechnology. Illustrative viewers include a web browser, a clientapplication, a mapping component, an imagery viewer, or the like. By wayof example, an illustrative imagery viewer includes the virtual-globeviewer “Google Earth” offered by Google Inc. of Mountain View, Calif.Imagery and other information can be rendered in mapping components andimagery viewers through commands indicated in a KML file. This file canbe static or delivered dynamically over the network.

At a step 1220, input 1222 is provided to server 1214. This takes theform of requesting a state update. Input 1222 could be any type of theinputs that we've previously mentioned, other types of inputs includinginput by a user or by another machine. This input is received at server1214, which might interact with dataset 1218 to, at a step 1224, updatea user interface on first viewer 1212 by providing information 1226. Inaccordance with an embodiment of our technology, this update request1220 might also propagate another update instance 1228 by providinginformation 1230 to second viewer 1216. Thus, information received byway of a first user interface might cause a change in that first userinterface and might also cause other changes to occur in other viewersthat are being run in connection with server 1214. For example, a usermight see a tile of interest in an imagery viewer, request more tileslike that tile, and those tiles will be updated in the imagery viewer,but also displayed in a grid such as grid 1142 in a web browser forexample.

Similarly, at a step 1232, input 1234 could be received from secondviewer 1216 and communicated to server 1214, which provokes an updateprocess 1224 and/or 1228.

Turning now to FIG. 13, an illustrative embodiment for a user interfacefor receiving query inputs is provided and referenced generally by thenumeral 1300. The various functional aspects offered by way of userinterface 1300 might be implemented in a variety of ways. We show a fewillustrative controls to help convey various aspects of our technology,but do not mean to imply that user interface 1300 is limiting in natureunless we specifically say otherwise.

In one embodiment, a drop-down menu 1310 allows a user to select a placeof interest. As illustratively shown, “Natanz, Iran,” is shown as havingbeen selected. Thus, in some embodiments, various places arepredetermined and exposed by way of a drop-down menu. In otherembodiments, a location can be entered by a user using a text box suchas text box 1314. By way of example, a user might enter a zip code, alatitude or longitude coordinate, or some other location indication aswe have described or that would be clear to a person of ordinary skillin the art.

A drop down menu 1312 enables a user to select a specific time period.For example, as shown by way of example, two dates (“2006-09-19 and2007-06-11”) have been selected. Control 1312 does not need to be adrop-down menu. Temporal information can be provided by way of a textbox or some other control.

In one embodiment, a text box 1314 is provided to enable a user to inputone or more geographic areas. It might be the case that a user is awareof indicia that identifies specific areas of interest. If that is thecase, then that indicia can be input into text box 1316. In anotherembodiment, a text box 1317 allows a CQM threshold or range to be inputto, for example, locate only geographic areas containing change resultsderived from imagery with a specific CQM constraint (e.g. “greater than7” or “between 7 and 9”)

An images area 1318 can also be provided. In one embodiment, this allowsa user to draw the shape of an item of interest in area 1318, whichmight actually be utilized to upload multiple images (such as a beforeimage and an after image). In other embodiments, it might receive ascreen capture of an image of interest that is to be used as the basisfor a query. And in still other embodiments, a user might wish to uploadimage pairs. This can be done by way of “upload” button 1320.

Still further, a change-intensity text box 1322 is provided to enable auser to indicate a desired change in intensity that is to be used inconnection with satisfying a given query. Attempting to list all of thedifferent types of ways that inputs can be provided might tend toobscure aspects of our technology. Suffice it to say controls can beprovided by way of a user interface such as that of 1300 to receive anyof the different types of inputs that we have described, or any of thetypes of constraints that will be levied by way of a query against ageospatial dataset.

Area 1324 shows three illustrative buttons that expose certainfunctional aspects of our technology. A “link to earth” button 1326 isprovided. Button 1326 enables a link to be established to anearth-viewing application. In some embodiments certain services areemployed, and links to a given service might need to be established.This can happen automatically in the background or by way of a button.

Button 1328 allows the application of feedback. As was previouslydescribed, and as will also be described below, feedback by way of auser or other entity can be applied to a first set of search results toderive a second or as many subsequent desired search results as desired.

A reset button 1330 can be used to quickly either wipe out the datainputs provided or return them to some default state. We have alsodescribed screen shot 1100 in FIG. 11A, which depicts other illustrativeways of receiving various types of geographic inputs that can be used toconstrain a query.

Turning now to FIG. 14A, an illustrative screen shot 1400 depicts a setof results that have been returned incident to processing a query. Wewill be providing examples to help illustrate aspects of the invention.By providing these examples, we do not mean to imply that our inventionis constrained by what might be depicted in connection with a specificexample. They are illustrative in nature.

Control 1410 indicates that the place of interest in this example isNatanz, Iran. The relevant time period is between Sep. 19, 2006 and Jun.11, 2007, as indicated by control 1412. If multiple dates were desiredto be searched, then this can be done and would be reflected in screenshot 1400 in one embodiment. As previously mentioned, no dates could beprovided, in which case, any change information associated with Natanz,Iran, would be returned to a user. In this example, a user is seekingchange information associated with a geographic area, Natanz, Iran,between the two dates mentioned.

After processing the query, a set of results are returned.Representations of those results are depicted in screen shot 1400. Aspreviously described, in one embodiment, the results are rankedaccording to their change score, which we variously refer to as adistance measurement, wherein we use the term “distance” not to indicatea physical distance measurement such as length or meters; rather, itindicates a relevancy. A legend 1414 indicates by way of illustrationthat some tiles returned might be of high relevance, some medium, somelow, and some not relevant at all. One can apply as many buckets here asis desired. When the tiles are ranked, one embodiment of our technologygroups them in sets. For example, they might be grouped in sets of 20.We show the first 500 ranked tiles in sets of twenty by way of numeral1416. That is, numeral 1416 indicates several groupings of tiles. Thefirst grouping indicates the first tiles, 1-20, which would be the 20most relevant tiles based on the inputs 1410 and 1412. Though this mightnot always be the case, the first set of ranked tiles are indicated ashigh, the middle set between medium and high, and then the remainingsets are indicated as having medium relevancy. If the geographic areacontains a great many regions that are very similar to the querycriteria, then it might be the case that several sets of the tiles areconsidered highly relevant.

Box 1418 indicates a given tile ranking. Box 1418 indicates a tileranking of “one,” that is, the geographic area associated with that tileis the geographic area that includes features that are most relevant tothe provided query. By way of illustration, this geographic area mighthave UTM coordinates as show in box 1419, e.g., “567080M E, 3731756M N.”This tile has been given a change score of “2.6” as indicated by Box1420.

We have previously described that user-feedback options can be presentedto a user, thereby enabling a reception of user-feedback indicationsthat indicate a desire to see images that are more alike or dislike agiven image. Illustrative user-feedback options are shown in box 1422.We show them as smiley faces to help convey to a user the concept ofdesire, neutral, and undesired options. These might also be referred toas a “more like this,” neutral, or “fewer like this” options. A useracting on the happy smiley face would convey that the user wishes toreceive more results that are similar to the results associated with thegiven geographic area. A user indicating the frowning face wouldindicate that the user does not want to see images that look like theimages associated with a given geographic area.

Image 1426 represents a given geographic area in a first state. Thismight be a “before” state. Image 1428 indicates the geographic area inan “after” state. If more states were available, then they could also beshown. That is, dataset 252 might include several images of the samegeographic area. If that is the case, then the multiple states could beshown here. Image 1430 is an example of a change map. Image 1430indicates change intensity associated with the two images. That is,image 1430 visually depicts an amount of change associated with a givengeographic area. Because this is a patent application, much of thedetail in pictures 1426 and 1428 is not presented. For example, image1426 might really correspond to a photograph, in which dirt and otherlow-level features would be shown. We show in image 1426 outlines ofmore prevalent items associated with the geographic area relevant toimage 1426. For example, one can think of the outlines as buildingstructures and the linear aspects as roads. Image 1428 indicates thesame geographic area but at a second state. We try to show by image 1430the concept of change intensity. The more densely packed dots representareas of greater change, which, again, is somewhat hard to glean basedon the line drawings of Images 1426 and 1428. But for purposes ofillustration, it can be seen in change map 1430 that certain areas ofthe given geographic area have changed more than others. A user's eye isquickly drawn to those areas of intense change. More than justconglomerating the dots as shown, they can also be color-coded. We havepreviously mentioned that in some ways, a change map such as that of1430 can relate to weather-radar map. If that were the case here, thecenter portion of the change map 1430 might be bright red, whereas theextremities might be light blue.

We show in FIG. 14B a copy of FIG. 14A, but which includes images 1426B,1428B, and 1430B, which are more detailed representations thatrespectively correspond to images 1426, 1428 and 1430 in FIG. 14.

Returning to FIG. 14A, box 1432 indicates the second most highly rankedtile, which has a chain score of “2.5” as indicated by box 1434, andalso includes user-feedback options 1436.

In FIG. 15, screen shot 1500 depicts even more groupings of ranked tiles1510, as well as successively lower ranked tiles. For example, the thirdmost highly ranked tile is shown by 1510, having a change score of “2.3”as indicated by 1514. Relevance-feedback can be provided by way of theoptions indicated by 1516. A change map 1518 depicts change between thetiles associated with the second geographic region. Change map 1520depicts change associated with the images of the third geographicregion. All of the items in 1510, as well as those of 1416 in FIG. 14A,include hyper links that, if acted on, push the respective set of tilesinto a viewer. We have previously mentioned that illustrative viewerscould include an imagery viewer, as well as a web browser, which isdepicted in FIG. 16.

FIG. 16 depicts 20 tiles in depiction 1600. In this example, the 20tiles all correspond to before images of each of their respectivegeographic regions. The most highly ranked tiles appear in the first rowin this embodiment. For example, image 1601 is the same image as 1426 inFIG. 14A. This enables a user to see a set of tiles together. The tilesmight be in some state, such as a first state, a second state, a thirdstate, etc.

An indication of the number of tiles is provided by way of reference1610, which indicates that tiles 1-20 are being presented.User-navigation controls are also provided. For example, arrow 1612would allow, if applicable, scrolling to bring into view a previous setof tiles, and then an initial set by way of control 1614. Similarly,arrow 1616 would present a next set of tiles, and control 1618 mightpresent a last set of tiles in one embodiment. The tiles can be cycled,as indicated by 1620. To cycle a tile means to rotate through differentstates of a geographic area. For example, assume tile 1601 correspondedto first geographic area 110 of FIG. 1B. If cycling were initiated, thenthe area that image 1601 occupies would variously cycle through firststate 114, then second state 116, then, if desired, change map 122. Inone embodiment all 20 tiles could cycle through their respective statesand change maps if desired.

FIG. 17A is a line drawing that indicates outlines of areas of changeassociated with respective geographic regions. That is, each tile ofdepiction 1700 corresponds to a certain geographic region. In FIG. 17A,20 change maps are presented. Screen shot 17B is a screen shot thatcorresponds to FIG. 17A, but shows in greater detail the gradientnatures of each change map. That is, FIG. 17B illustrates areas withinthe outlines of FIG. 17A. In one embodiment of an actual implementation,the outlines of FIG. 17B are not present. The outlines are shown herebecause of constraints associated with drafting patent applications.But, although they could be shown in some embodiments, in otherembodiments the actual outlines of FIG. 17B are not present, only theinteriors are presented.

FIG. 18 depicts an illustrative presentation 1800 of a set of tiles inan “after” state. Thus, presentation 1800 is similar to that of 1600 inFIG. 16, except after images of the respective geographic areas arepresented instead of “before” images. We also illustratively show thatuser-feedback options could be presented in connection with each image.By way of example, 1810 refers to a set of user-feedback options.Specifically, 1812 refers to an illustrative user-feedback option thatenables the user to indicate a desire to receive more tiles like theinstant tile. Option 1814 indicates a neutral indication. Option 1816indicates an illustrative user-feedback option for enabling a user toindicate that the instant tile does not contain features that a userdesires to see and/or contains features that a user does not desire tosee. Clearly, other types of user-feedback options besides smiley facescould be included. An implementer might present check boxes, a slider,or any number of controls that are adapted to receive user-feedbackindications that indicate a preference associated with one or moreimages. The user-feedback from multiple images can be internalized andprocessed at the same time, as previously discussed. See, for example,FIGS. 8G-8I. Incident to receiving as many user-feedback indications asindicated, a revised set of tiles can be presented.

For example, FIG. 19 depicts a presentation 1900 that illustrates that auser has indicated a desire to see tiles that are unlike tile 1924 (byselecting option 1910), unlike tile 1922 (by way of option 1912), moretiles like tile 1918 (by way of option 1914), and more tiles like 1920(indicated by option 1916).

Incident to processing the user-feedback of FIG. 19, presentation 2000of FIG. 20 can be rendered in a viewer. FIG. 20 accounts for the twonegative-feedback indications and the two positive-feedback indicationsto present the tiles that are shown. The new tiles rankings are based onsimilarity to the positive-feedback tiles and dissimilarity to thenegative-feedback tiles. These new tile rankings can be dynamicallypushed to another viewer application, such as an earth imagery viewer.Referring back to FIG. 19, one observes that tiles 1918 and 1920exhibited some linear characteristics. Presentation 2000 depicts othertiles that also share features such as having linear features. The tilespresented in presentation 2000 represent the “after” state of ageographic area. These also could be cycled, or toggled to represent a“before” state.

Turning now to FIG. 21, an illustrative imagery viewer is provided andreferenced generally by 2100. A depiction of the earth 2112 is presentedin one embodiment. Reference 2110 indicates that we will continue ourexample that we previously started explaining in connection to Natanz,Iran. A network link has been established to imagery viewer 2100,resulting in the tree structure that is being depicted in area 2114.

Suppose that a user wishes to push a set of tiles into the imageryviewer. For example, with reference to FIG. 14A, assume that a userwishes to push the first 20 tiles of 1416 into an imagery viewer usingKML. This could be accomplished by clicking on the hyper linkedgrouping, which results in a depiction illustratively shown in FIG. 22and referenced generally by 2200. Reference 2210 indicates that theplace of interest is Natanz, Iran (see 1410) and 2212 indicates thedates corresponding to the dates denoted by reference 1412 in FIG. 14.User interface 2213 allows these tiles (variously referred to as“chips”) to be presented in context. The various tiles appear in thepatent drawing as black squares, but this is not how they would appearin real life in one embodiment. For example, they might appear asoutlines, and depict a state of the geographic region. We show them asfilled-in black so that they would be easier to refer to at thiszoomed-out stage. By way of example, 2216 and 2214 correspond to twotiles. Reference 2218 refers to a set of seven tiles, and 2220 refers tofive more tiles. Finally, reference 2222 refers to another five tiles.These 20 tiles are the same 20 tiles that were initially presented inFIG. 14A, as well as FIG. 16.

It can now be seen that areas of change associated with a very largearea are highlighted so that a user can focus in on those specificgeographic areas associated with each of the tiles. We are depictingrelevant data. That is, determined as relevant to a query that a user,or other entity, provided. Rather than having to search the entiregeographic area 2213, a user can now view specific areas of change.

Absent our technology, a user would be faced with searching geographicarea 2213, and particularly searching geographic area 2213 in a givenstate. That is, a user might not even be aware that certain areas havechanged over time. A user might be able to find an item of initialinterest, such as a road, but that item might not be of particularinterest because the user would not know that that road was recentlyinstalled. A recently installed road might indicate a development ofsome sorts which development might be relevant for some reason. Forexample, say that a state in the United States was prohibited fromcreating some large development in some area. The presence of a roadmight indicate the beginnings of such a development.

Turning now to FIG. 23, a presentation 2300 depicts tiles 2222 (fromFIG. 22) in greater detail. The tiles are associated with the followingreference numerals 2310, 2312, 2314, 2316, and 2318. These are five ofthe tiles that appeared in FIG. 22. We are showing them somewhat zoomedin to illustrate other aspects of our technology. We note that from thedrawings it appears as though tiles 2310-2318 appear to merely beoutlines. But really, each tile reflects a depiction of background 2320in various states. For example, the tiles, collectively referred to bythe numeral 2322, could depict a first state of each respectivegeographic region, a second state of the applicable geographic region,intermediate states, as well as change maps associated with changebetween any of the aforementioned states. In the screenshots thatfollow, we will illustrate by way of line drawings the reality thattiles 2322 depict information, and are not merely outlines.

Background 2320 allows tiles 2322 to be viewed in context. Tiles 2322are tiles of specific geographic areas, such as those viewed on one ormore of the previous referred to grids. But if a user wishes to observethe tiles in the midst of their respective surroundings, then pushingthe tiles out to an imagery viewer such as that shown enables such acontextual viewing. Each tile includes interaction options in oneembodiment. By way of illustration, tiles 2322 each include what appearsto be a bull's eye in the center of each tile. This bull's eye providesa way for a user to interact with each of the tiles. In alternativeembodiments, outlining could be provided, the outlining could behyperlinked, or the tile itself could be hyperlinked. We will nowdescribe illustrative aspects of interacting with the tiles, startingwith FIG. 24.

FIG. 24 depicts an illustrative presentation 2400 that includes tiles2419, which can be the same tiles 2322 that appeared in FIG. 23. Wefurther show a control 2412 that exposes a plurality of interactionoptions. Three illustrative options are shown within control 2412, butothers are available as well. We indicate that others are available byway of placeholder object 2420, which is shown in greater detail byreference numeral 2422, which illustrates three other links (2424, 2426,and 2428), which might only be a subset of still more links, illustratedby the ellipses. In certain aspects of the drawings we show an ellipsesto clarify that there are other options besides those shown, but we donot mean to imply that in drawings that do not include an ellipses thatthe options shown are the only options. Sometimes, different aspects ofthe different drawings might tend to appear as though certain optionsmight be exhausted. Rarely any are, but sometimes we show an ellipses tohelp make clear that certain options that might otherwise seem topossibly indicate an exhausted list that such a list is not exhausted.

Control 2412 includes three links: A small link 2414, a large link 2416,and a smart link 2418. In the embodiment being described, each of theselinks relates to a surround function. That is, small link 2414, if actedon, would surround a given tile by a set of tiles determined to bewithin some radius. For example, perhaps small link 2414 would causeevery contiguous tile to be presented that touches a given tile.

Similarly, large link 2416 enables a presentation of tiles that arewithin a greater radius than those presented by way of small link 2414.These options can be useful when a user wishes to view change associatedwith surrounding areas that surround a given tile. In some embodiments,such change in such tiles might not have been sufficiently high to havebeen ranked among the set that is currently in view. These optionsprovide a way to view change information associated with nearby tiles.

Smart-surround link 2418 applies a little more intelligence than smalllink 2414 or large link 2416. The smart-surround option 2418 analyzesnearby tiles to determine which of those might have a change patternsimilar to that of a given tile. In one embodiment, if nearby tiles havechanged beyond some amount, they are pushed to screenshot 2400. In otherembodiments, a window of parameters is applied such that change thatfalls within such a window is used to return geographic areas.

Other links such as those in 2422 can be used to expose other functionalaspects. For example, link 2424 might be used to locate other geographicareas that appear similar to a given tile, and provide a representationof that tile. In some embodiments, the representation takes the form ofan arrow pointing in a direction that leads to the target, or found,tile. In other embodiments, the tile is brought into view by way of anoverlay or popup that appears on top of screenshot 2400. In still otherembodiments, the user interface flies or tours to the target tile,thereby bringing it into view. Thus, for example, assume that a personis interested in tile 2422. If a user wants to find more tiles that looklike say a before state of region 2422, then acting on link 2424 mightautomatically bring a target tile into view, provided that geographicarea found includes features similar to the features of a before stateof tile 2422. Similarly, searching could be done based on an after stateof region 2422. Still further, another link, such as link 2426, mightbring into view tiles that had changed in a manner similar to the waythat geographic area 2422 has changed. Acting on such a link 2426 couldautomatically bring into view that tile, or a set of tiles.

Other options, such as link 2428 might be used to add a given geographicarea to a set of favorites, such that a list is maintained of tiles ofinterest. The tiles that make up such a list can also be interactedwith, such that they can serve as inputs to an individual query to findmore geographic areas that exhibit features shared by the geographicareas added to the list.

Turning now to FIG. 25, a screenshot 2500 depicts the results of a userhaving acted on smart-surround link 2418 of FIG. 24. Now, tiles 2514 areshown. In FIG. 24, it was difficult to tell that a roadway feature mightbe viewable. Let us assume that tile 2510 corresponds to tile 2422 ofFIG. 24. The upper left hand corner of tile 2510 includes what now canbe seen as a roadway feature. Because the smart-surround option 2418 wasacted on, other tiles have been brought into view, which more clearlymakes visible roadway structure 2512. For example, tile 2516 is nowvisible. Tile 2516 might not have been visible in FIG. 24 because,although it does depict a portion of roadway structure 2512, it mightnot have exhibited enough change to be sufficiently highly ranked to beinitially presented in presentation 2400. But by clicking thesmart-surround option associated with tile 2510, our technology receivesan indication of the key associated with tile 2510, and then searchesthrough, in one embodiment, the list of ranked tiles to determine othertiles that have changed and that are nearby to tile 2510. As a result ofthis analysis, tiles such as tile 2518 and 2520 are pushed out topresentation 2500, thereby enabling a viewing or cycling of states ofthe corresponding geographic regions. That is, each of the tiles mightpresent a before state, an after state, or a change map. Although sometiles are shown by mere outlines without any fill patterns, we show themin this manner to illustrate the concept that they can be retrieved, butleave them blank so as to highlight aspects of this specific example,wherein roadway 2512 is identified.

Returning now to FIG. 23, a user can drill down and zoom in on tiles2322 of FIG. 23, and then interact with a tile such as tile 2422 toreceive even more information about nearby geographic areas that havechanged in a manner similar to tile 2422. Acting on such an option 2418,yields tiles 2514 in FIG. 25, which clearly indicate the presence of anew road structure 2512. Thus, one embodiment of our technology allowsthe rapid searching of a large area, such as Natanz, Iran, to veryrapidly identify a new road structure such as road structure 2512. Now,a user can analyze the surrounding geographic area 2524 in greaterdetail. A user can analyze why such a new road 2512 has been recentlybuilt. The applications that are made available by way of our technologyare legion. A user is now armed with an ability to quickly locate areasof interest. This is a benefit over and above merely depicting theentire geographic area of Natanz all at once. That is, our technologydoes not require presenting huge amounts of data, which would otherwisetake a long amount of time and large amount of space to present in aviewer application such as that depicted by presentation 2500.

In contrast, we expose the ability to automatically and quickly detectgeographic change and present these results efficiently and rapidly to auser. While this specific example illustrated the use of locating anewly developed road, the examples are legion for determining geographicareas that have changed in many different ways. Thus, if othergeographic areas are sought that exhibit change characteristics similarto those of any of the change areas depicted in a viewer as a result ofa user query, than these can be easily gathered and presented as well.

Turning now to FIG. 26 presentation 2600 illustrates another example offurther presenting a set of options associated with a given tile, suchas tile 2612, which is among tiles 2608. Acting on a smart-surroundoption 2610 might yield depiction 2700 of FIG. 27, which indicates apresentation of tiles 2712, 2714, 2716, 2718, 2720, 2722 and tile 2724based on having interacted with tile 2710, which might have been thesame tile as 2612 of FIG. 26. Now, roadway 2730 is even more clearlybrought into view.

To recapitulate, our technology enables a user to rapidly locategeographic areas of interest that have changed in a certain manner. Wehave shown, for example, where a user inputs a query into a webinterface to look for areas that have changed using various constraintssuch as location, time, change intensity, CQM, etc. (FIG. 14). Theweb-interface communicates over a network to a server which retrievesthe relevant information from a database (FIG. 13) and returns to theweb interface a ranked list of results (FIG. 14.) satisfying the query.The user reviews these results in the web interface in a variety ofvisualization formats (FIGS. 15 and 16), which include animations thatshow the tiles in a before and after state and that also may include achange intensity map (FIG. 17). In one embodiment of a web viewer, theuser provides positive feedback on tiles that have changed in a mannerof interest to the user, such as a new road being constructed, andnegative feedback on tiles that have changed in a manner of no interest,such as agricultural areas of change (FIGS. 18 and 19). These resultsare communicated back from the web-interface to the server, whichanalyzes the feedback and retrieves a revised set of change regionsbased on the feedback. The revised results are viewed in the web viewer(FIG. 20) and then pushed to an earth imagery viewer for contextualvisualization (FIGS. 21 and 22). In the earth viewer, the user interactswith the regions of interest and through additional tile-level controlsis able to expand the area of interest using various options (FIGS. 23,24, 25, 26, 27). These controls communicate back to the server whichretrieves additional areas of interest from the database and then pushesthese back into the earth imagery viewer. Finally, the earth imageryviewer also has controls that allow the before and after states of theimagery to be cycled.

We have previously mentioned that the tiles can be cycled in states.Turning to FIG. 28, presentation 2800 depicts a control 2810 thatenables a user to set a time 2814 by way of control 2812 that allows oneor more of tiles 2816 to automatically cycle states. That is, tiles 2816might be animated. Consider a specific tile, such as tile 2818.According to one aspect of our technology, tile 2818 can cycle through.That is, a first state associated with geographic area 2818 can bepresented, and then another state, and then another state, and thenperhaps the change map, etc. The dates can be constrained by control2812. By way of example, Sep. 8, 2007 is shown. If a beginning time weredesired, it can also be provided. Thus, those tiles falling with anygiven date range would cycle through. Although this is unable to beshown in a stagnate document, a viewer viewing presentation 2800 wouldsee certain tiles automatically changing states in an animated fashionwithout user interaction. This even more clearly directs a user'sattention to areas of change, and allows a user to view first states,intermediate states, and ending states as well as change maps in acyclical pattern.

Although we have described aspects associated with presenting changeinformation associated with the geographic area, we will nowrecapitulate various aspects in connection with FIGS. 29A and 29B.Turning first to FIG. 29A, an illustrative method for presenting aspectsof change associated with a given geographic area is provided generallyby the numeral 2910. At a step 2912, a query is received that reflects arequest to present a visualization of terrestrial change associated witha given geographic area. In one embodiment, the query includes one ormore inputs. The inputs might include location-identificationinformation, time-span information, or a first and second image. In thecase where inputs are first and second images, feature information canbe extracted from the images on demand in one embodiment. In otherembodiments, this feature information has already been extracted and ispre-stored prior to receiving the query. The inputs might also include achange-quality measurement as previously described. As previouslymentioned, a dataset, such as dataset 252 (see FIG. 1) can be searchedto satisfy the query. Such a database 252 might include heterogeneouscross-referencing data that facilitates cross referencing geospatialdata with other information to return desired results.

At a step 2914, the stored dataset is inspected. In one embodiment, thestored dataset includes indexed geospatial data that are usable to besearched to respond to the query. The indexed geospatial data includeindexed information that describes the geographic area in oneembodiment. As described, the indexed information could have beenderived from high-resolution, remotely sensed imagery. The indexedinformation can include information that describes items within ageographic area as well as a geographic area itself. Illustrative itemsthat might be within a geographic area include things like airplanes,baseball diamonds, buildings, and other man-made or evennaturally-occurring items.

At a step 2916, regions that satisfy the query are identified from thedataset. With reference to FIG. 1B, by way of illustration, illustrativeregions that might have been identified include first geographic area110 or second geographic area 112.

At a step 2918, ranking values can be assigned to each region. We havepreviously described various methods of assigning ranking values toidentify the regions. The ranking values indicate a relevancy score suchthat, in one embodiment, those regions having a higher ranking valuethan others are more relevant than those having lower ranking values. Ata step 2920, presentations respectively associated with the rankedregions are presented on a viewer. We have explained that a viewer cantake on various forms including a web page, mapping application, ongeospatial mapping platform, which might be a part of or separate from avirtual-globe presentation program.

In one embodiment, the presentations presented include an animation thatincludes sequencing images of the geographic area that are depicted invarious states. We have described illustrative processes in connectionwith, for example, FIG. 28. With reference to FIG. 22, an illustrativedepiction 2200 illustrates presenting a set of ranked regions on a userinterface. They could also be presented in a grid-like pattern, such asis shown in FIG. 16 for example. The presentations might include anindication of the ranked regions. Thus, by way of example, wordsdescribed in the regions might be presented. In other embodiments,pictures associated with the regions might be presented, as was shown inFIG. 14B for example. FIG. 14A also indicates hyperlinked rankings 1416,which can also be included in the presentations. The presentations mightalso include a visual depiction of change. We show an example of this inFIG. 14A, with reference to numeral 1430. The presentation might alsoinclude a “before” image that corresponds to a first state of thegeographic area. For example, the first state might be first state 114or first state 130 of FIG. 1B. Similarly, the presentation might includean “after” image that corresponds to a second state of a givengeographic area. In FIG. 1B, we show examples of two second states 116,132, which respectively correspond to two geographic areas. Of course,as many intermediate images as are available can be presented.

The presentation might also include an identified area that has changedin a manner similar to how a given area has changed. That is, a firstgeographic area might be presented. This geographic area for sake ofexample, includes imagery that describes a first state as well asimagery that describes a second state. From these two images, changeinformation can be identified in the applicable dataset. This changeinformation can then be used to search the dataset for other geographicareas that are associated with similar change information. To be similarto the change information might contemplate a similar change intensity.In other embodiments, similarity might include similarity to a givenfeature band. For example, say the reference area exhibited certainchange characteristics in the red spectral band. Perhaps a user wishesto search on other geographic areas that exhibit similar change in thered spectral band without concern about change in other bands. By way ofanother example, perhaps the reference or source geographic areaexhibited certain textural-related change. Because, as mentioned, weindex feature and change information at the band level, we can search ona per band basis. Similarly, we can search on a set of bands whileignoring other bands. Still further, perhaps change detection is desiredto be temporarily constrained. In this case, items that have changedover a similar period of time could be particularly retrieved andpresented.

As previously described, the presentations might be presented as part ofa larger geographic area such that they are visually distinguishablewithin an overall presentation that includes the larger geographic area.With reference to FIG. 22, larger geographic area 2213 provides abackground upon which to present the various tiles that are shown, suchas, for example, tiles 2218.

At a step 2922, selectable links that are associated with a presentedregion can be provided. We show some illustrative links in FIG. 24.

Turning now to FIG. 29B, another illustrative method for presentingaspects of change associated with various geographic areas is provided,and referenced generally by the numeral 2930. In one embodiment, at astep 2932, this method includes receiving a query that is directed at acertain geographic area. The query seeks to identify regions that arecharacterized by aspects of change based on inputs to the query. We havepreviously described an array of types of inputs that can be provided inconnection with the query. At a step 2934, the query is applied to adataset of geospatial information that stores imagery associated withthe geographic area. This dataset includes information that issufficient to identify the regions.

At a step 2936, a first results set was received that includes a firstset of keys and corresponding change scores (which we variously refer toas difference or distance scores). By way of FIG. 8F, we describe someillustrative instances wherein keys associated with various rankingscores are returned. Keys can take on any format as long as they areable to uniquely identify geographic areas. The change score indicatesan amount of change in the given region.

At a step 2938, the results are visualized in a viewing application. Wehave previously mentioned that all or a portion of the results set caninclude transitioning images that depict various states of the certainregion, which can also include cycling against a background so that theimages are looped.

Turning now to FIG. 30A, an illustrative process for revising a firstset of search results is provided and referenced generally by thenumeral 3010. At a step 3012, a first set of search results areretrieved incident to receiving a query that requests a set ofgeographic areas that are associated with certain changecharacteristics. The query can include various options that describedesired aspects of the set of geographic areas that are sought by way ofthe query.

At a step 3014, a first set of images is presented based on the query.By way of example, FIG. 19 depicts an illustrative example of presentinga first set of search results. At a step 3016, a user-feedback option ispresented in connection with each image. We do not mean to imply that auser-feedback option needs to be presented with every image shown onFIG. 19. To the extent that some images do not include user-feedbackoptions, those might not be considered within the aforementioned set.Illustrative user-feedback options are shown in FIG. 19 by way of smileyfaces, which have been previously described. The happy faces indicatepositive feedback, while the frowning faces indicate negative feedback.Thus, in one embodiment, at a step 3018, the smiling faces areuser-feedback options that allow reception of the user-feedbackindication that indicates a desire to receive more results that aresimilar to the image associated with the given smiling face. Similarly,a frowning face is an illustrative example of a user-feedback optionthat enables reception of a user-feedback indication that indicates adesire to receive more results that are dissimilar to the imageassociated with the frowning face. As previously mentioned, an array ofother types of controls could be utilized instead of smiley faces, whichare provided to illustrate the concept.

At a step 3020, a second set of images are retrieved based on the userfeedback. In one embodiment, the second set of images were derived fromthe first set of images, but in other embodiments, the second set ofimages are either wholly or in part separate from the first set ofimages.

At a step 3022, the second set of images are presented on a viewer. Theviewer might be the same viewer that was used to present the first setof search results or a different viewer.

Turning now to FIG. 30B, another illustrative method for revising afirst set of search results is provided and referenced generally by thenumber 3030. At a step 3032, a query is received that seeks a first setof images that depict geographic areas that are characterized by acertain set of features. We have previously described an array of typesof features that can be searched on. For example, see FIG. 7E. Aninitial set of images based on the query is returned at a step 3034.User-feedback indications are received but each indicates arelevance-feedback score that is attributable to images in the initialset of images at a step 3036. At a step 3038, a second set of images isreturned based on the user-feedback indications.

In one embodiment, the images that are returned were done so based onchange information associated with the corresponding geographic areas.

Turning now to FIG. 30C, still another embodiment for revising a firstset of search results is provided and referenced generally by thenumeral 3040. At a step 3042, a query is received that seeks geographicareas that have changed consistent with the given change signature.Thus, in this embodiment, change information is provided as an input toa query, and the first set of results that are returned are thoseresults that are associated with change information that is similar tochange information associated with one or more geographic areas uponwhich the query is based.

At a step 3044, initial indications of the geographic areas arereturned. At a step 3046, a first set of images corresponding to theindications returned in step 3044 are presented.

At a step 3048, feedback options are presented in connection with eachimage, such that a second set of images can be identified based onreceiving input by way of the feedback options. The feedback optionsmight include “more like this” or “less like this options,” where eachoption indicates a desire to receive more geographic areas that aresimilar to a certain region or regions, or more that are dissimilar tothe given region or regions.

Many different arrangements of the various components depicted, as wellas components not shown, are possible without departing from the spiritand scope of this disclosure. Embodiments have been described with theintent to be illustrative rather than restrictive. Alternativeembodiments will become apparent to those skilled in the art uponreading this disclosure.

It will be understood that certain features and sub-combinations are ofutility and may be employed without reference to other features andsub-combinations and are contemplated within the scope of the claims.Not all steps listed in the various figures need be carried out in thespecific order described.

1. One or more computer-readable media having computer-useableinstructions for performing a method of presenting aspects of changeassociated with a geographic area that has been captured byhigh-resolution, remotely sensed imagery, the method comprising:receiving a query that reflects a request to present a visualization ofterrestrial change associated with the geographic area, wherein thequery includes one or more inputs; inspecting a stored dataset thatincludes indexed geospatial data that is useable to be searched torespond to the query, (1) wherein the indexed geospatial data includesindexed information that describes the geographic area, and (2) whereinthe indexed information was derived from the high-resolution, remotelysensed imagery; identifying from the dataset a first plurality ofregions that satisfy the query, wherein each of the first plurality ofregions, (1) is a portion of a larger geographic area, and (2) reflectschange in the portion from a first state to a second state; respectivelyassigning a ranking value to each region of the first plurality ofregions, wherein the ranking value is useable to rank the firstplurality of regions based on their relative consistency with the query,thereby providing a set of ranked regions; and presenting on a userinterface one or more presentations respectively associated with aselection of the ranked regions.
 2. The media of claim 1, wherein theone or more inputs include one or more of: location-identificationinformation; time-span information; and a first image and a secondimage, wherein feature information is extracted from the first andsecond images on demand, and becomes part of the stored dataset.
 3. Themedia of claim 2, wherein the one or more inputs further includes achange-quality measurement that indicates a suitability of comparing twoimages.
 4. The media of claim 3, wherein the change-quality measurementis a measure of likelihood that the terrestrial change reflectshuman-induced change.
 5. The media of claim 2, wherein the datasetfurther includes heterogeneous cross-referencing data that facilitatescross referencing the geospatial data with other information to returndesired results, and wherein the other information includes indicationsof locations of items of interest.
 6. The media of claim 5, wherein theitems of interest include one or more of: types of organizations;geographic features; transportation features; populated areas; andmilitary establishments.
 7. The media of claim 1, wherein the indexedinformation that describes the geographic area includes information thatdescribes items within the geographic area.
 8. The media of claim 1,wherein the one or more presentations includes an animation thatincludes sequencing images of the geographic area that depict it invarious states including the first and second states.
 9. The media ofclaim 8, wherein the one or more presentations further includes one ormore of: an indication of one or more of the ranked regions; a visualdepiction of change within at least one of the ranked regions; a beforeimage corresponding to the first state; an after image corresponding tothe second state; and an identified area that has changed in a mannersimilar to how a reference area has changed.
 10. The media of claim 8,wherein the visual depiction of change includes a change-confidenceimage that indicates, areas of change of a given geographic region, andan intensity of change across the areas.
 11. The media of claim 10,wherein the intensity of change is on a per-pixel basis such that eachpixel in the change-confidence image reflects an intensity of changebetween two respective pixels in two images, which includes a firstimage that reflects the given geographic region in a first state and asecond image that reflects the geographic region in a second state. 12.The media of claim 9, wherein the one or more presentations arepresented as part of the larger geographic area such that the one ormore presentations are visually distinguishable within an overallpresentation that includes the larger geographic area.
 13. The media ofclaim 12, wherein the user interface includes a viewer, which viewerincludes one or more of: a web page; a mapping component; and ageospatial mapping platform including a virtual-globe presentationprogram.
 14. The media of claim 13, wherein the viewer presents abackground upon which to present the one or more presentations, suchthat the selection of the ranked regions are placed in visual context byway of the background.
 15. The media of claim 12, further comprisingproviding one or more selectable links associated with each of thepresented regions of the selection, wherein the one or more selectablelinks includes one or more of the following: a first link that exposes afirst surround option that presents change attributes on the backgroundfor a first series of regions that are contiguous with a certain regionthat is associated with the one or more selectable links, wherein thefirst series of regions are presented irrespective of an amount of theirchange information; a second link that exposes a second surround optionthat presents change attributes on the background for a second series ofregions that are within a predetermined proximity to the given certainregion, wherein the second series of regions are presented irrespectiveof an amount of their change information; a third link that exposes athird surround option that presents change attributes on the backgroundfor a third series of regions that are within a given proximity to thegiven certain region, wherein the third series of regions are presentedonly if they are associated with a given threshold level of change; afourth link that exposes a fourth surround option that presents changeattributes on the background for a fourth series of regions that arewithin a given proximity to the given certain region, wherein the fourthseries of regions are presented only if they are associated with asimilar level of change as that of the certain region; a fifth link thatfacilitates identification of a first set of one or more other regionssimilar to the given region, and automatically brings into view thefirst set of one or more other regions; and a sixth link thatfacilitates identification of a second set of one or more other regionsthat have changed in a manner similar to the way that the given regionhas changed, and automatically brings into view the second set of one ormore other regions.
 16. One or more computer-readable media havingcomputer-useable instructions for performing a method of presentingaspects of change associated with a geographic area that has beencaptured by high-resolution, remotely sensed imagery, the methodcomprising: receiving a query directed at the geographic area thatincludes one or more inputs, the query seeking an identification ofregions associated with the geographic area that are characterized byaspects of change based on the one or more inputs; applying the query toa dataset of geospatial information that stores imagery associated withthe geographic area, wherein the dataset includes information that issufficient to identify the regions, and wherein the imagery is derivedfrom the high-resolution, remotely sensed imagery, which ischaracterized by having a resolution of three meters or less per pixel;receiving a first results set that includes a first plurality of keysand corresponding change scores, wherein, (1) each key is useable toidentify a certain region, and (2) each change score indicates an amountof change in the certain region from a first state to a second state;presenting at least a portion of the results set in a viewingapplication.
 17. The media of claim 89, wherein the presenting at leasta portion of the results set includes transitioning images that depictthe first state and the second state of the certain region.
 18. Themedia of claim 17, wherein the transitioning includes cycling thatappears against a background that includes the geographic area.
 19. Oneor more computer-readable media having computer-useable instructions forperforming a method of presenting aspects of a geographic area that hasbeen captured by high-resolution, remotely sensed imagery, the methodcomprising: receiving a query that reflects a request to present avisualization of the aspects of the geographic area, wherein the queryincludes one or more inputs; inspecting a stored dataset that includesindexed geospatial data that is useable to be searched to respond to thequery, (1) wherein the indexed geospatial data includes indexedinformation that describes the geographic area, and (2) wherein theindexed information was derived from the high-resolution, remotelysensed imagery; identifying from the dataset a first plurality ofregions that satisfy the query, wherein each of the first plurality ofregions is a portion of a larger geographic area; respectively assigninga ranking value to each region of the first plurality of regions,wherein the ranking value is useable to rank the first plurality ofregions based on their relative consistency with the query, therebyproviding a set of ranked regions; and presenting on a user interfaceone or more presentations respectively associated with a selection ofthe ranked regions; and presenting a link associated with a given regionof the ranked regions, wherein the link, if acted on, presents anindication of another region that has features in common with the givenregion.
 20. The media of claim 19, wherein the identifying from thedataset a first plurality of regions includes identifying one or moreitems associated with one or more of the first plurality of regions. 21.The media of claim 20, wherein the indication of the another regionincludes one or more of: the another region; and a directionalidentifier that points to the another region.